Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review

OBJECTIVES

Conversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions.

MATERIALS AND METHODS

We conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO).

RESULTS

The review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages.

DISCUSSION AND CONCLUSION

This review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research.

PROTOCOL REGISTRATION

The Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.

A Virtual Assistant to Guide Early Postoperative Rehabilitation after Reverse Shoulder Arthroplasty: A Pilot Randomized Trial

INTRODUCTION

Rehabilitation can improve outcomes after reverse shoulder arthroplasty (RSA). However, low adherence to rehabilitation and compliance rates are some of the main barriers. To address this public health issue, the goal of this research was to pilot test and evaluate the effectiveness of a chatbot to promote adherence to home rehabilitation in patients undergoing RSA.

METHODS

A randomized pilot trial including patients undergoing RSA and early postoperative rehabilitation was performed. The control group received standard home rehabilitation; the experimental group received the same intervention supervised with a chatbot, with automated interactions that included messages to inform, motivate, and remember the days and exercises for 12 weeks. Compliance with rehabilitation and clinical measures of shoulder function, pain, and quality of life were assessed.

RESULTS

31 patients (17 experimental) with an average age of 70.4 (3.6) completed the intervention. Compliance was higher in the experimental group (77% vs. 65%; OR95% = 2.4 (0.5 to 11.4)). Statistically significant between-group differences with a CI of 95% were found in the QuickDASH questionnaire and self-reported quality of life. No differences were found in the rest of the measures.

CONCLUSIONS

This pilot study suggests that the chatbot tool can be useful in promoting compliance with early postoperative home rehabilitation in patients undergoing RSA. Future randomized trials with adequate power are warranted to determine the clinical impact of the proposal.

Effects of a Gamified Agent-Based System for Personalized Elderly Care: Pilot Usability Study

BACKGROUND

The global percentage of older people has increased significantly over the last decades. Information and communication technologies have become essential to develop and motivate them to pursue healthier ways of living. This paper examines a personalized coaching health care service designed to maintain living conditions and active aging among older people. Among the technologies the service includes, we highlight the use of both gamification and cognitive assistant technologies designed to support older people and an application combining a cognitive virtual assistant to directly interact with the older person and provide feedback on their current health condition and several gamification techniques to motivate the older person to stay engaged with the application and pursuit of healthier daily habits.

OBJECTIVE

This pilot study aimed to investigate the feasibility and usability of a gamified agent-based system for older people and obtain preliminary results on the effectiveness of the intervention regarding physical activity health outcomes.

METHODS

The study was designed as an intervention study comparing pre- and posttest results. The proposed gamified agent-based system was used by 12 participants over 7 days (1 week), and step count data were collected with access to the Google Fit application programming interface. Step count data after the intervention were compared with average step count data before the intervention (average daily values over a period of 4 weeks before the intervention). A　1-tailed Student t test was used to determine the relationship between the dependent and independent variables. Usability was measured using the System Usability Scale questionnaire, which was answered by 8 of the 12 participants in the study.

RESULTS

The posttest results showed significant pre- to posttest changes (P=.30; 1-tailed Student t test) with a moderate effect size (Cohen d=0.65). The application obtained an average usability score of 78.

CONCLUSIONS

The presented pilot was validated, showing the positive health effects of using gamification techniques and a virtual cognitive assistant. Additionally, usability metrics considered for this study confirmed high adherence and interest from most participants in the pilot.

Evaluation of the Rosa Chatbot Providing Genetic Information to Patients at Risk of Hereditary Breast and Ovarian Cancer: Qualitative Interview Study

BACKGROUND

Genetic testing has become an integrated part of health care for patients with breast or ovarian cancer, and the increasing demand for genetic testing is accompanied by an increasing need for easy access to reliable genetic information for patients. Therefore, we developed a chatbot app (Rosa) that is able to perform humanlike digital conversations about genetic BRCA testing.

OBJECTIVE

Before implementing this new information service in daily clinical practice, we wanted to explore 2 aspects of chatbot use: the perceived utility and trust in chatbot technology among healthy patients at risk of hereditary cancer and how interaction with a chatbot regarding sensitive information about hereditary cancer influences patients.

METHODS

Overall, 175 healthy individuals at risk of hereditary breast and ovarian cancer were invited to test the chatbot, Rosa, before and after genetic counseling. To secure a varied sample, participants were recruited from all cancer genetic clinics in Norway, and the selection was based on age, gender, and risk of having a BRCA pathogenic variant. Among the 34.9% (61/175) of participants who consented for individual interview, a selected subgroup (16/61, 26%) shared their experience through in-depth interviews via video. The semistructured interviews covered the following topics: usability, perceived usefulness, trust in the information received via the chatbot, how Rosa influenced the user, and thoughts about future use of digital tools in health care. The transcripts were analyzed using the stepwise-deductive inductive approach.

RESULTS

The overall finding was that the chatbot was very welcomed by the participants. They appreciated the 24/7 availability wherever they were and the possibility to use it to prepare for genetic counseling and to repeat and ask questions about what had been said afterward. As Rosa was created by health care professionals, they also valued the information they received as being medically correct. Rosa was referred to as being better than Google because it provided specific and reliable answers to their questions. The findings were summed up in 3 concepts: "Anytime, anywhere"; "In addition, not instead"; and "Trustworthy and true." All participants (16/16) denied increased worry after reading about genetic testing and hereditary breast and ovarian cancer in Rosa.

CONCLUSIONS

Our results indicate that a genetic information chatbot has the potential to contribute to easy access to uniform information for patients at risk of hereditary breast and ovarian cancer, regardless of geographical location. The 24/7 availability of quality-assured information, tailored to the specific situation, had a reassuring effect on our participants. It was consistent across concepts that Rosa was a tool for preparation and repetition; however, none of the participants (0/16) supported that Rosa could replace genetic counseling if hereditary cancer was confirmed. This indicates that a chatbot can be a well-suited digital companion to genetic counseling.

Health Information From Web Search Engines and Virtual Assistants About Pre-Exposure Prophylaxis for HIV Prevention in Adolescents and Young Adults: Content Analysis

BACKGROUND

Adolescents and young adults are disproportionately affected by HIV, suggesting that HIV prevention methods such as pre-exposure prophylaxis (PrEP) should focus on this group as a priority. As digital natives, youth likely turn to internet resources regarding health topics they may not feel comfortable discussing with their medical providers. To optimize informed decision-making by adolescents and young adults most impacted by HIV, the information from internet searches should be educational, accurate, and readable.

OBJECTIVE

The aims of this study were to compare the accuracy of web-based PrEP information found using web search engines and virtual assistants, and to assess the readability of the resulting information.

METHODS

Adolescent HIV prevention clinical experts developed a list of 23 prevention-related questions that were posed to search engines (Ask.com, Bing, Google, and Yahoo) and virtual assistants (Amazon Alexa, Microsoft Cortana, Google Assistant, and Apple Siri). The first three results from search engines and virtual assistant web references, as well as virtual assistant verbal responses, were recorded and coded using a six-tier scale to assess the quality of information produced. The results were also entered in a web-based tool determining readability using the Flesch-Kincaid Grade Level scale.

RESULTS

Google web search engine and Google Assistant more frequently produced PrEP information of higher quality than the other search engines and virtual assistants with scores ranging from 3.4 to 3.7 and 2.8 to 3.3, respectively. Additionally, the resulting information generally was presented in language at a seventh and 10th grade reading level according to the Flesch-Kincaid Grade Level scale.

CONCLUSIONS

Adolescents and young adults are large consumers of technology and may experience discomfort discussing their sexual health with providers. It is important that efforts are made to ensure the information they receive about HIV prevention methods, and PrEP in particular, is comprehensive, comprehensible, and widely available.

"Alexa, did the pandemic make you smarter?" A follow up content analysis of a virtual assistant's responses to a prenatal query

To compare responses to 40 common prenatal questions from Amazon's virtual assistant, Alexa, one year apart during the COVID pandemic. Participants: Two researchers replicated a prenatal query using unique Alexa devices. A conceptual content analysis was conducted where the researchers independently queried Alexa the identical questions from their 2020 study during the same one-week timeframe, between May 20, 2021 and May 27, 2021. Alexa's responses were compared to the 2020 study and the American College of Obstetricians and Gynecologists data and verified by one of the researchers, a Certified Nurse Midwife. Alexa provided accurate responses to 26 (65%) of the questions, an increase by 55 percentage points from 2020. Alexa was able to recite the symptoms of COVID-19 illness but was unable to provide a response to the two other COVID-specific questions. Compared to the 2020 query, Alexa provided more reputable sources for the responses including the CDC, WHO, NIH, and Mayo Clinic. Alexa's ability to provide more accurate, evidence-based responses was remarkably improved in 2021. Mobile health tools, like Amazon Alexa, are highly utilized by the public, particularly with limited healthcare access during the COVID-19 pandemic. Technology-based platforms should provide credible, evidence-based content.

Transforming Medical Education: Assessing the Integration of ChatGPT Into Faculty Workflows at a Caribbean Medical School

INTRODUCTION

ChatGPT is a Large Language Model (LLM) which allows for natural language processing and interactions with users in a conversational style. Since its release in 2022, it has had a significant impact in many occupational fields, including medical education. We sought to gain insight into the extent and type of usage of ChatGPT at a Caribbean medical school, the American University of Antigua College of Medicine (AUA).

METHODS

We administered a questionnaire to 87 full-time faculty at the school via email. We quantified and made graphical representations of the results via Qualtrics Experience Management software (QualtricsXM, Qualtrics, Provo, UT). Survey results were investigated using bar graph comparisons of absolute numbers and percentages for various categories related to ChatGPT usage, and descriptive statistics for Likert scale questions.

RESULTS

We found an estimated 33% of faculty were currently using ChatGPT. There was broad acceptance of the program by those who were using it and most believed it should be an option for students. The primary task ChatGPT was being used for was multiple choice question (MCQ) generation. The primary concern faculty had was incorrect information being included in ChatGPT output.

CONCLUSION

ChatGPT has been quickly adopted by a subset of the college faculty, demonstrating its growing acceptance. Given the level of approval expressed about the program, we believe ChatGPT will continue to form an important and expanding part of faculty workflows at AUA and in medical education in general.

The Effects of a Health Care Chatbot's Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study

BACKGROUND

The rising adoption of telehealth provides new opportunities for more effective and equitable health care information mediums. The ability of chatbots to provide a conversational, personal, and comprehendible avenue for learning about health care information make them a promising tool for addressing health care inequity as health care trends continue toward web-based and remote processes. Although chatbots have been studied in the health care domain for their efficacy for smoking cessation, diet recommendation, and other assistive applications, few studies have examined how specific design characteristics influence the effectiveness of chatbots in providing health information.

OBJECTIVE

Our objective was to investigate the influence of different design considerations on the effectiveness of an educational health care chatbot.

METHODS

A 2×3 between-subjects study was performed with 2 independent variables: a chatbot's complexity of responses (eg, technical or nontechnical language) and the presented qualifications of the chatbot's persona (eg, doctor, nurse, or nursing student). Regression models were used to evaluate the impact of these variables on 3 outcome measures: effectiveness, usability, and trust. A qualitative transcript review was also done to review how participants engaged with the chatbot.

RESULTS

Analysis of 71 participants found that participants who received technical language responses were significantly more likely to be in the high effectiveness group, which had higher improvements in test scores (odds ratio [OR] 2.73, 95% CI 1.05-7.41; P=.04). Participants with higher health literacy (OR 2.04, 95% CI 1.11-4.00, P=.03) were significantly more likely to trust the chatbot. The participants engaged with the chatbot in a variety of ways, with some taking a conversational approach and others treating the chatbot more like a search engine.

CONCLUSIONS

Given their increasing popularity, it is vital that we consider how chatbots are designed and implemented. This study showed that factors such as chatbots' persona and language complexity are two design considerations that influence the ability of chatbots to successfully provide health care information.

What do individuals with visual impairment need and want from a dialogue-based digital assistant?

CLINICAL SIGNIFICANCE

Optometrists are well-placed to provide helpful advice and guidance to patients with visual impairment but may not know how best to do this. The availability of a reliable and comprehensive conversational agent to which patients could be directed would be a valuable supplement to clinical intervention.

BACKGROUND

The Artificial Intelligence in Visual Impairment (AIVI) Study is a proof-of-concept study to investigate whether ongoing information support for people with visual impairment (VI) can be provided by a dialogue-based digital assistant. The phase of the AIVI Study reported here explored the different dimensions of the information-seeking behaviour of individuals with VI: in particular, their need for information, the methods for obtaining it at present, and their views on the use of a digital assistant.

METHODS

Qualitative data were collected from 120 UK-resident adults who responded to an online survey who were either visually impaired (86.7%), a carer or family member of someone with VI (5.8%), or a professional involved in the support of those with VI (7.5%). In addition, 10 in-depth 1:1 semi-structured interviews explored opinions in more detail. Thematic analysis was used to analyse the findings.

RESULTS

Analysis of information needs identified 7 major themes: ocular condition; equipment, technology and adaptations; daily activities; registration; finance/employment; emotional support; and support for the carer. Participants used a wide variety of methods to access information from many sources and explained the barriers to access. Participants accepted the merit of a dialogue system aiding in a goal-directed search for specific information, but expressed reservations about its abilities in other areas, such as providing emotional support.

CONCLUSIONS

Participants highlighted potential benefits, limitations, and requirements in using a digital assistant to access information about VI. These findings will inform the design of dialogue systems for populations with VI.

Machine Learning Algorithms for Detection and Classifications of Emotions in Contact Center Applications

Over the past few years, virtual assistant solutions used in Contact Center systems are gaining popularity. One of the main tasks of the virtual assistant is to recognize the intentions of the customer. It is important to note that quite often the actual intention expressed in a conversation is also directly influenced by the emotions that accompany that conversation. Unfortunately, scientific literature has not identified what specific types of emotions in Contact Center applications are relevant to the activities they perform. Therefore, the main objective of this work was to develop an Emotion Classification for Machine Detection of Affect-Tinged Conversational Contents dedicated directly to the Contact Center industry. In the conducted study, Contact Center voice and text channels were considered, taking into account the following families of emotions: anger, fear, happiness, sadness vs. affective neutrality of the statements. The obtained results confirmed the usefulness of the proposed classification—for the voice channel, the highest efficiency was obtained using the Convolutional Neural Network (accuracy, 67.5%; precision, 80.3; F1-Score, 74.5%), while for the text channel, the Support Vector Machine algorithm proved to be the most efficient (accuracy, 65.9%; precision, 58.5; F1-Score, 61.7%).

Convenient Access to Expert-Reviewed Health Information via an Alexa Voice Assistant Skill for Patients With Multiple Myeloma: Development Study

Background

Patients with multiple myeloma (MM) have high information needs due to the complexity of the disease and variety of treatments. Digital voice assistants provide support in daily life and can be a convenient tool that even older patients can use to access health information. Voice assistants may therefore be useful in providing digital health services to meet the information needs of patients with MM.

Objective

We aim to describe and report on the development, content, and functionality of the first Amazon Alexa voice assistant skill for patients with MM in Germany with the goal of empowering and educating patients. Further, we share data on skill usage and first learnings.

Methods

In a cocreation workshop with MM patient organizations and MM medical experts in Germany, Takeda Oncology discussed the development and content of the Alexa skill Multiple Myeloma. Patient information on MM disease, diagnostics, and therapy was presented in a question-and-answer format, reviewed by experts, and programmed into the skill. Additionally, a search function for finding patient support groups within a perimeter of 200 km around the users and a myeloma quiz functionality with multiple-choice questions were integrated into the skill. Aggregated retrospective data on the total number of skill installations and skill usage were retrieved from an Amazon Alexa developer account, and a web-based patient survey was conducted on the Takeda Oncology website.

Results

The Alexa skill Multiple Myeloma was launched in September 2019. It was available free of charge on the German Amazon Alexa skill store between September 2019 and March 2022 and could be used with devices featuring the Amazon Alexa voice assistant. Since the launch in September 2019 and up to July 2021, a total of 141 users have installed the skill. Between July 2020 and July 2021, a total of 189 skill sessions with 797 utterances were analyzed. The most popular inquiries were searches for patient support groups near the users (58/797, 7.3%), followed by inquiries about information on MM disease (53/797, 6.6%) and the quiz (43/797, 5.4%). The web-based survey on voice assistant usage and the feedback on the Alexa skill Multiple Myeloma were collected from 24 participants and showed that 46% (11/24) of participants would recommend the Alexa skill. Nonusers of voice assistants (11/24, 46%) stated that data protection concerns (7/11, 64%) and a lack of need (6/11, 55%) were the most important factors of not using voice assistants.

Conclusions

The Alexa skill Multiple Myeloma offers patient-friendly and expert-reviewed answers and explanations for medical terms related to MM disease, diagnostics, and therapy, as well as connections to patient support groups and a quiz functionality. In the future, the skill can be extended with new content and functionalities, such as medication adherence support.

A Deep Learning-Based Chinese Semantic Parser for the Almond Virtual Assistant

Almond is an extendible open-source virtual assistant designed to help people access Internet services and IoT (Internet of Things) devices. Both are referred to as skills here. Service providers can easily enable their devices for Almond by defining proper APIs (Application Programming Interfaces) for ThingTalk in Thingpedia. ThingTalk is a virtual assistant programming language, and Thingpedia is an application encyclopedia. Almond uses a large neural network to translate user commands in natural language into ThingTalk programs. To obtain enough data for the training of the neural network, Genie was developed to synthesize pairs of user commands and corresponding ThingTalk programs based on a natural language template approach. In this work, we extended Genie to support Chinese. For 107 devices and 261 functions registered in Thingpedia, 649 Chinese primitive templates and 292 Chinese construct templates were analyzed and developed. Two models, seq2seq (sequence-to-sequence) and MQAN (multiple question answer network), were trained to translate user commands in Chinese into ThingTalk programs. Both models were evaluated, and the experiment results showed that MQAN outperformed seq2seq. The exact match, BLEU, and F1 token accuracy of MQAN were 0.7, 0.82, and 0.88, respectively. As a result, users could use Chinese in Almond to access Internet services and IoT devices registered in Thingpedia.

Hands-Free Authentication for Virtual Assistants with Trusted IoT Device and Machine Learning

Virtual assistants, deployed on smartphone and smart speaker devices, enable hands-free financial transactions by voice commands. Even though these voice transactions are frictionless for end users, they are susceptible to typical attacks to authentication protocols (e.g., replay). Using traditional knowledge-based or possession-based authentication with additional invasive interactions raises users concerns regarding security and usefulness. State-of-the-art schemes for trusted devices with physical unclonable functions (PUF) have complex enrollment processes. We propose a scheme based on a challenge response protocol with a trusted Internet of Things (IoT) autonomous device for hands-free scenarios (i.e., with no additional user interaction), integrated with smart home behavior for continuous authentication. The protocol was validated with automatic formal security analysis. A proof of concept with websockets presented an average response time of 383 ms for mutual authentication using a 6-message protocol with a simple enrollment process. We performed hands-free activity recognition of a specific user, based on smart home testbed data from a 2-month period, obtaining an accuracy of 97% and a recall of 81%. Given the data minimization privacy principle, we could reduce the total number of smart home events time series from 7 to 5. When compared with existing invasive solutions, our non-invasive mechanism contributes to the efforts to enhance the usability of financial institutions' virtual assistants, while maintaining security and privacy.

Smart Solutions to Keep Your Mental Balance

Due to the coronavirus pandemic international conflicts, dramatic changes of daily living have been enforced, including new ways of providing patient assistance, based on artificial intelligence. The influence of these changes on people's mental health is still insufficiently analyzed and explored. Chatbots like Woebot, Wysa and Tess are gaining popularity, being attractive and easy to use. These achievements led us to develop a new application, being still in the testing phase, which has a positive impact on mental healthcare issues. It is a conversational system capable to diagnose people's negative, depressive, and anxious emotions during chatting, and to act as a psychological therapist and virtual friend. The proposed system, throughout the conversation, succeeds to decrease the patient's insecurity sentiments, by comforting their mood. In fact, an intelligent assistant for different mental health issues like stress, anxiety and depression, could become a very helpful information system.

Improving User Experience of Virtual Health Assistants: Scoping Review

Background

Virtual assistants can be used to deliver innovative health programs that provide appealing, personalized, and convenient health advice and support at scale and low cost. Design characteristics that influence the look and feel of the virtual assistant, such as visual appearance or language features, may significantly influence users’ experience and engagement with the assistant.

Objective

This scoping review aims to provide an overview of the experimental research examining how design characteristics of virtual health assistants affect user experience, summarize research findings of experimental research examining how design characteristics of virtual health assistants affect user experience, and provide recommendations for the design of virtual health assistants if sufficient evidence exists.

Methods

We searched 5 electronic databases (Web of Science, MEDLINE, Embase, PsycINFO, and ACM Digital Library) to identify the studies that used an experimental design to compare the effects of design characteristics between 2 or more versions of an interactive virtual health assistant on user experience among adults. Data were synthesized descriptively. Health domains, design characteristics, and outcomes were categorized, and descriptive statistics were used to summarize the body of research. Results for each study were categorized as positive, negative, or no effect, and a matrix of the design characteristics and outcome categories was constructed to summarize the findings.

Results

The database searches identified 6879 articles after the removal of duplicates. We included 48 articles representing 45 unique studies in the review. The most common health domains were mental health and physical activity. Studies most commonly examined design characteristics in the categories of visual design or conversational style and relational behavior and assessed outcomes in the categories of personality, satisfaction, relationship, or use intention. Over half of the design characteristics were examined by only 1 study. Results suggest that empathy and relational behavior and self-disclosure are related to more positive user experience. Results also suggest that if a human-like avatar is used, realistic rendering and medical attire may potentially be related to more positive user experience; however, more research is needed to confirm this.

Conclusions

There is a growing body of scientific evidence examining the impact of virtual health assistants’ design characteristics on user experience. Taken together, data suggest that the look and feel of a virtual health assistant does affect user experience. Virtual health assistants that show empathy, display nonverbal relational behaviors, and disclose personal information about themselves achieve better user experience. At present, the evidence base is broad, and the studies are typically small in scale and highly heterogeneous. Further research, particularly using longitudinal research designs with repeated user interactions, is needed to inform the optimal design of virtual health assistants.

Empowering Dementia Carers With an iSupport Virtual Assistant (e-DiVA) in Asia-Pacific Regional Countries: Protocol for a Pilot Multisite Randomized Controlled Trial

Background

Dementia is a global public health priority with an estimated prevalence of 150 million by 2050, nearly two-thirds of whom will live in the Asia-Pacific region. Dementia creates significant care needs for people with the disease, their families, and carers. iSupport is a self-help platform developed by the World Health Organization (WHO) to provide education, skills training, and support to dementia carers. It has been adapted in some contexts (Australia, India, the Netherlands, and Portugal). Carers using the existing adapted versions have identified the need to have a more user-friendly version that enables them to identify solutions for immediate problems quickly in real time. The iSupport virtual assistant (iSupport VA) is being developed to address this gap and will be evaluated in a randomized controlled trial (RCT).

Objective

This paper reports the protocol of a pilot RCT evaluating the iSupport VA.

Methods

Seven versions of iSupport VA will be evaluated in Australia, Indonesia, New Zealand, and Vietnam in a pilot RCT. Feasibility, acceptability, intention to use, and preliminary impact on carer-perceived stress of the iSupport VA intervention will be assessed.

Results

This study was funded by the e-ASIA Joint Research Program in November 2020. From January to July 2023, we will enroll 140 dementia carers (20 carers per iSupport VA version) for the pilot RCT. The study has been approved by the Human Research Committee, University of South Australia, Australia (203455).

Conclusions

This protocol outlines how a technologically enhanced version of the WHO iSupport program—the iSupport VA—will be evaluated. The findings from this intervention study will provide evidence on the feasibility and acceptability of the iSupport VA intervention, which will be the basis for conducting a full RCT to assess the effectiveness of the iSupport VA. The study will be an important reference for countries planning to adapt and enhance the WHO iSupport program using digital health solutions.

Trial Registration

Australian New Zealand Clinical Trials Registry ACTRN12621001452886; https://tinyurl.com/afum5tjz

International Registered Report Identifier (IRRID)

PRR1-10.2196/33572

Conversational Agents for Health and Well-being Across the Life Course: Protocol for an Evidence Map

BACKGROUND

Conversational agents, which we defined as computer programs that are designed to simulate two-way human conversation by using language and are potentially supplemented with nonlanguage modalities, offer promising avenues for health interventions for different populations across the life course. There is a lack of open-access and user-friendly resources for identifying research trends and gaps and pinpointing expertise across international centers.

OBJECTIVE

Our aim is to provide an overview of all relevant evidence on conversational agents for health and well-being across the life course. Specifically, our objectives are to identify, categorize, and synthesize-through visual formats and a searchable database-primary studies and reviews in this research field.

METHODS

An evidence map was selected as the type of literature review to be conducted, as it optimally corresponded to our aim. We systematically searched 8 databases (MEDLINE; CINAHL; Web of Science; Scopus; the Cochrane, ACM, IEEE, and Joanna Briggs Institute databases; and Google Scholar). We will perform backward citation searching on all included studies. The first stage of a double-stage screening procedure, which was based on abstracts and titles only, was conducted by using predetermined eligibility criteria for primary studies and reviews. An operational screening procedure was developed for streamlined and consistent screening across the team. Double data extraction will be performed with previously piloted data collection forms. We will appraise systematic reviews by using A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2. Primary studies and reviews will be assessed separately in the analysis. Data will be synthesized through descriptive statistics, bivariate statistics, and subgroup analysis (if appropriate) and through high-level maps such as scatter and bubble charts. The development of the searchable database will be informed by the research questions and data extraction forms.

RESULTS

As of April 2021, the literature search in the eight databases was concluded, yielding a total of 16,351 records. The first stage of screening, which was based on abstracts and titles only, resulted in the selection of 1282 records of primary studies and 151 records of reviews. These will be subjected to second-stage screening. A glossary with operational definitions for supporting the study selection and data extraction stages was drafted. The anticipated completion date is October 2021.

CONCLUSIONS

Our wider definition of a conversational agent and the broad scope of our evidence map will explicate trends and gaps in this field of research. Additionally, our evidence map and searchable database of studies will help researchers to avoid fragmented research efforts and wasteful redundancies. Finally, as part of the Harnessing the Power of Conversational e-Coaches for Health and Well-being Through Swiss-Portuguese Collaboration project, our work will also inform the development of an international taxonomy on conversational agents for health and well-being, thereby contributing to terminology standardization and categorization.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/26680.

Medication Adherence Reminder System for Virtual Home Assistants: Mixed Methods Evaluation Study

Background

Medication nonadherence is a global public health challenge that results in suboptimal health outcomes and increases health care costs. Forgetting to take medicines is one of the most common reasons for unintentional medication nonadherence. Research findings indicate that voice-activated virtual home assistants, such as Amazon Echo and Google Home devices, may be useful in promoting medication adherence.

Objective

This study aims to create a medication adherence app (skill), MedBuddy, for Amazon Echo devices and measure the use, usability, and usefulness of this medication-taking reminder skill.

Methods

A single-group, mixed methods, cohort feasibility study was conducted with women who took oral contraceptives (N=25). Participants were undergraduate students (age: mean 21.8 years, SD 6.2) at an urban university in the Southeast United States. Participants were given an Amazon Echo Dot with MedBuddy—a new medication reminder skill for Echo devices created by our team—attached to their study account, which they used for 60 days. Participants self-reported their baseline and poststudy medication adherence. MedBuddy use was objectively evaluated by tracking participants’ interactions with MedBuddy through Amazon Alexa. The usability and usefulness of MedBuddy were evaluated through a poststudy interview in which participants responded to both quantitative and qualitative questions.

Results

Participants’ interactions with MedBuddy, as tracked through Amazon Alexa, only occurred on half of the study days (mean 50.97, SD 29.5). At study end, participants reported missing their medication less in the past 1 and 6 months compared with baseline (χ²₁=0.9 and χ²₁=0.4, respectively; McNemar test: P<.001 for both). However, there was no significant difference in participants’ reported adherence to consistently taking medication within the same 2-hour time frame every day in the past 1 or 6 months at the end of the study compared with baseline (χ²₁=3.5 and χ²₁=0.4, respectively; McNemar test: P=.63 and P=.07, respectively). Overall feedback about usability was positive, and participants provided constructive feedback about the skill’s features that could be improved. Participants’ evaluation of MedBuddy’s usefulness was overwhelmingly positive—most (15/23, 65%) said that they would continue using MedBuddy as a medication reminder if provided with the opportunity and that they would recommend it to others. MedBuddy features that participants enjoyed were an external prompt separate from their phone, the ability to hear the reminder prompt from a separate room, multiple reminders, and verbal responses to prompts.

Conclusions

The findings of this feasibility study indicate that the MedBuddy medication reminder skill may be useful in promoting medication adherence. However, the skill could benefit from further usability enhancements.

On the Security and Privacy Challenges of Virtual Assistants

Since the purchase of Siri by Apple, and its release with the iPhone 4S in 2011, virtual assistants (VAs) have grown in number and popularity. The sophisticated natural language processing and speech recognition employed by VAs enables users to interact with them conversationally, almost as they would with another human. To service user voice requests, VAs transmit large amounts of data to their vendors; these data are processed and stored in the Cloud. The potential data security and privacy issues involved in this process provided the motivation to examine the current state of the art in VA research. In this study, we identify peer-reviewed literature that focuses on security and privacy concerns surrounding these assistants, including current trends in addressing how voice assistants are vulnerable to malicious attacks and worries that the VA is recording without the user’s knowledge or consent. The findings show that not only are these worries manifold, but there is a gap in the current state of the art, and no current literature reviews on the topic exist. This review sheds light on future research directions, such as providing solutions to perform voice authentication without an external device, and the compliance of VAs with privacy regulations.

Virtual Home Assistant Use and Perceptions of Usefulness by Older Adults and Support Person Dyads

AIM

Describe virtual home assistant use and usefulness from the perspective of older adults and their support persons.

METHODS

This was a mixed-methods study with older adults and their support persons (n = 10 dyads). Virtual home assistant (VHA) equipment was installed in participants' homes, and its use was documented for 60 days. Participants received protocol-guided telephone calls to address their VHA questions or problems. The type and frequency of VHA use were summarized with descriptive statistics. End-of-study interviews about VHA use were conducted with dyad participants. Qualitative content analyses were used to describe the interview findings about the dyad's perceptions of using the VHA, how it was used, any difficulties experienced, and suggestions for future VHA uses.

RESULTS

Participant dyads reported positive VHA perceptions, including the potential for VHAs to promote aging in place. Participants discussed the challenges learning the technology and replacing old habits with new ones. Participants offered recommendations for future VHA skills and for more education and training about using the VHA.

CONCLUSIONS

The study findings suggest that VHAs may be useful for older adults as they age in place and offer reassurance for support persons.

The Alexafication of Adult Social Care: Virtual Assistants and the Changing Role of Local Government in England

Voice controlled virtual assistants, delivered via consumer devices such as smart speakers and tablets, are being trialled by local authorities across England as a convenient and low-cost supplement or potential alternative to "traditional" telecare. Few papers have explored this increasingly widespread phenomenon, despite its growing importance. This article looks at choices by some local authorities to trial Alexa, within the context of the ongoing care crisis in England, with councils facing depleted funds, a lack of expert guidance on care technologies, and an increasingly complex and fragmented care technology marketplace. It draws on interviews with managers from eight English local authorities involved in the commissioning and trialling of technologies for adult social care to examine how and why virtual assistants are being implemented, and what implications their use might hold for care. Scaling up the application of such technologies could shift the role of local authorities towards one of an app developer and data broker, while generating considerable risks of reliance on the precarious technological infrastructure of global corporations that may have little interest in or sensitivity towards local care concerns. The findings suggest an urgent need for a national social care technology strategy and increased support for local authorities.

Monitoring Health Parameters of Elders to Support Independent Living and Improve Their Quality of Life

Improving the well-being and quality of life of the elderly population is closely related to assisting them to effectively manage age-related conditions such as chronic illnesses and anxiety, and to maintain their independence and self-sufficiency as much as possible. This paper presents the design, architecture and implementation structure of an adaptive system for monitoring the health and well-being of the elderly. The system was designed following best practices of the Human-Centred Design approach involving representative end-users from the early stages.

Clinical Advice by Voice Assistants on Postpartum Depression: Cross-Sectional Investigation Using Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana

BACKGROUND

A voice assistant (VA) is inanimate audio-interfaced software augmented with artificial intelligence, capable of 2-way dialogue, and increasingly used to access health care advice. Postpartum depression (PPD) is a common perinatal mood disorder with an annual estimated cost of $14.2 billion. Only a small percentage of PPD patients seek care due to lack of screening and insufficient knowledge of the disease, and this is, therefore, a prime candidate for a VA-based digital health intervention.

OBJECTIVE

In order to understand the capability of VAs, our aim was to assess VA responses to PPD questions in terms of accuracy, verbal response, and clinically appropriate advice given.

METHODS

This cross-sectional study examined four VAs (Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana) installed on two mobile devices in early 2020. We posed 14 questions to each VA that were retrieved from the American College of Obstetricians and Gynecologists (ACOG) patient-focused Frequently Asked Questions (FAQ) on PPD. We scored the VA responses according to accuracy of speech recognition, presence of a verbal response, and clinically appropriate advice in accordance with ACOG FAQ, which were assessed by two board-certified physicians.

RESULTS

Accurate recognition of the query ranged from 79% to 100%. Verbal response ranged from 36% to 79%. If no verbal response was given, queries were treated like a web search between 33% and 89% of the time. Clinically appropriate advice given by VA ranged from 14% to 29%. We compared the category proportions using the Fisher exact test. No single VA statistically outperformed other VAs in the three performance categories. Additional observations showed that two VAs (Google Assistant and Microsoft Cortana) included advertisements in their responses.

CONCLUSIONS

While the best performing VA gave clinically appropriate advice to 29% of the PPD questions, all four VAs taken together achieved 64% clinically appropriate advice. All four VAs performed well in accurately recognizing a PPD query, but no VA achieved even a 30% threshold for providing clinically appropriate PPD information. Technology companies and clinical organizations should partner to improve guidance, screen patients for mental health disorders, and educate patients on potential treatment.

Attitudes and Perceptions Toward Voice-Operated Smart Speakers Among Low-Income Senior Housing Residents: Comparison of Pre- and Post-Installation Surveys

Smart speakers have the potential to support independent living and wellness among low-income senior housing (LISH) residents. The aim of this study was to examine and compare LISH residents’ attitudes and perceptions toward smart speakers at two time points: before and after technology use (N = 47). A descriptive survey was administered to ask questions about hedonic motivation, perceived ease of use, self-efficacy, perceived usefulness of some potential or existing smart speaker features, cost, and privacy. Participants were initially favorable toward using a smart speaker and its digital agent (e.g., Alexa) as a daily assistant and wellness tool. They especially liked the smart speaker’s potential functionality of detecting harmful events and notifying someone to receive immediate help. The comparison of pre- and post-use responses revealed non-significant declines in most items, with the exception of willingness to use Alexa as a reminder system (p < .01), asking Alexa for help (p < .01), and asking for help in using Alexa (p < .01). This finding may reflect confusion or frustration with the device among participants. We conclude with recommendations for the design of smart speakers specifically tailored to the needs of LISH residents.

A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis

AI virtual assistants have significant potential to alleviate the pressure on overly burdened healthcare systems by enabling patients to self-assess their symptoms and to seek further care when appropriate. For these systems to make a meaningful contribution to healthcare globally, they must be trusted by patients and healthcare professionals alike, and service the needs of patients in diverse regions and segments of the population. We developed an AI virtual assistant which provides patients with triage and diagnostic information. Crucially, the system is based on a generative model, which allows for relatively straightforward re-parameterization to reflect local disease and risk factor burden in diverse regions and population segments. This is an appealing property, particularly when considering the potential of AI systems to improve the provision of healthcare on a global scale in many regions and for both developing and developed countries. We performed a prospective validation study of the accuracy and safety of the AI system and human doctors. Importantly, we assessed the accuracy and safety of both the AI and human doctors independently against identical clinical cases and, unlike previous studies, also accounted for the information gathering process of both agents. Overall, we found that the AI system is able to provide patients with triage and diagnostic information with a level of clinical accuracy and safety comparable to that of human doctors. Through this approach and study, we hope to start building trust in AI-powered systems by directly comparing their performance to human doctors, who do not always agree with each other on the cause of patients’ symptoms or the most appropriate triage recommendation.

The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review

Background

The high demand for health care services and the growing capability of artificial intelligence have led to the development of conversational agents designed to support a variety of health-related activities, including behavior change, treatment support, health monitoring, training, triage, and screening support. Automation of these tasks could free clinicians to focus on more complex work and increase the accessibility to health care services for the public. An overarching assessment of the acceptability, usability, and effectiveness of these agents in health care is needed to collate the evidence so that future development can target areas for improvement and potential for sustainable adoption.

Objective

This systematic review aims to assess the effectiveness and usability of conversational agents in health care and identify the elements that users like and dislike to inform future research and development of these agents.

Methods

PubMed, Medline (Ovid), EMBASE (Excerpta Medica dataBASE), CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Science, and the Association for Computing Machinery Digital Library were systematically searched for articles published since 2008 that evaluated unconstrained natural language processing conversational agents used in health care. EndNote (version X9, Clarivate Analytics) reference management software was used for initial screening, and full-text screening was conducted by 1 reviewer. Data were extracted, and the risk of bias was assessed by one reviewer and validated by another.

Results

A total of 31 studies were selected and included a variety of conversational agents, including 14 chatbots (2 of which were voice chatbots), 6 embodied conversational agents (3 of which were interactive voice response calls, virtual patients, and speech recognition screening systems), 1 contextual question-answering agent, and 1 voice recognition triage system. Overall, the evidence reported was mostly positive or mixed. Usability and satisfaction performed well (27/30 and 26/31), and positive or mixed effectiveness was found in three-quarters of the studies (23/30). However, there were several limitations of the agents highlighted in specific qualitative feedback.

Conclusions

The studies generally reported positive or mixed evidence for the effectiveness, usability, and satisfactoriness of the conversational agents investigated, but qualitative user perceptions were more mixed. The quality of many of the studies was limited, and improved study design and reporting are necessary to more accurately evaluate the usefulness of the agents in health care and identify key areas for improvement. Further research should also analyze the cost-effectiveness, privacy, and security of the agents.

International Registered Report Identifier (IRRID)

RR2-10.2196/16934

A Physical Activity and Diet Program Delivered by Artificially Intelligent Virtual Health Coach: Proof-of-Concept Study

Background

Poor diet and physical inactivity are leading modifiable causes of death and disease. Advances in artificial intelligence technology present tantalizing opportunities for creating virtual health coaches capable of providing personalized support at scale.

Objective

This proof of concept study aimed to test the feasibility (recruitment and retention) and preliminary efficacy of physical activity and Mediterranean-style dietary intervention (MedLiPal) delivered via artificially intelligent virtual health coach.

Methods

This 12-week single-arm pre-post study took place in Adelaide, Australia, from March to August 2019. Participants were inactive community-dwelling adults aged 45 to 75 years, recruited through news stories, social media posts, and flyers. The program included access to an artificially intelligent chatbot, Paola, who guided participants through a computer-based individualized introductory session, weekly check-ins, and goal setting, and was available 24/7 to answer questions. Participants used a Garmin Vivofit4 tracker to monitor daily steps, a website with educational materials and recipes, and a printed diet and activity log sheet. Primary outcomes included feasibility (based on recruitment and retention) and preliminary efficacy for changing physical activity and diet. Secondary outcomes were body composition (based on height, weight, and waist circumference) and blood pressure.

Results

Over 4 weeks, 99 potential participants registered expressions of interest, with 81 of those screened meeting eligibility criteria. Participants completed a mean of 109.8 (95% CI 1.9-217.7) more minutes of physical activity at week 12 compared with baseline. Mediterranean diet scores increased from a mean of 3.8 out of 14 at baseline, to 9.6 at 12 weeks (mean improvement 5.7 points, 95% CI 4.2-7.3). After 12 weeks, participants lost an average 1.3 kg (95% CI –0.1 to –2.5 kg) and 2.1 cm from their waist circumference (95% CI –3.5 to –0.7 cm). There were no significant changes in blood pressure. Feasibility was excellent in terms of recruitment, retention (90% at 12 weeks), and safety (no adverse events).

Conclusions

An artificially intelligent virtual assistant-led lifestyle-modification intervention was feasible and achieved measurable improvements in physical activity, diet, and body composition at 12 weeks. Future research examining artificially intelligent interventions at scale, and for other health purposes, is warranted.

Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review

Background

Conversational agents (also known as chatbots) have evolved in recent decades to become multimodal, multifunctional platforms with potential to automate a diverse range of health-related activities supporting the general public, patients, and physicians. Multiple studies have reported the development of these agents, and recent systematic reviews have described the scope of use of conversational agents in health care. However, there is scarce research on the effectiveness of these systems; thus, their viability and applicability are unclear.

Objective

The objective of this systematic review is to assess the effectiveness of conversational agents in health care and to identify limitations, adverse events, and areas for future investigation of these agents.

Methods

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols will be used to structure this protocol. The focus of the systematic review is guided by a population, intervention, comparator, and outcome framework. A systematic search of the PubMed (Medline), EMBASE, CINAHL, and Web of Science databases will be conducted. Two authors will independently screen the titles and abstracts of the identified references and select studies according to the eligibility criteria. Any discrepancies will then be discussed and resolved. Two reviewers will independently extract and validate data from the included studies into a standardized form and conduct quality appraisal.

Results

As of January 2020, we have begun a preliminary literature search and piloting of the study selection process.

Conclusions

This systematic review aims to clarify the effectiveness, limitations, and future applications of conversational agents in health care. Our findings may be useful to inform the future development of conversational agents and promote the personalization of patient care.

International Registered Report Identifier (IRRID)

PRR1-10.2196/16934

Development of a Complex Intervention to Improve Adherence to Antidiabetic Medication in Older People Using an Anthropomorphic Virtual Assistant Software

Introduction: Improving adherence to antidiabetic medication is crucial, resulting in improved health outcomes, cost reduction, and minimization of waste. A lack of underlying theory in existing interventions may explain the limited success in sustaining behavior change. This paper describes the development of a theory and evidence-based complex intervention to improve adherence to oral antidiabetics in older people via a software prototype with an anthropomorphic virtual assistant.

Methods: The Behavior Change Wheel (BCW) was used to develop a theoretical understanding of the change process, corresponding to the first phase of the Medical Research Council Framework for developing and evaluating complex interventions. At the BCW core is a model of human behavior (COM-B), which posits that human behavior (B) results from the interaction between capabilities (C), opportunities (O), and motivation (M). Literature-derived medication adherence determinants were mapped onto COM-B components. Then, intervention functions (IFs) were selected employing the APEASE criteria. Finally, standardized behavior change techniques (BCTs) were chosen based on their suitability and their effectiveness on medication adherence trials. The prototype was developed for android devices; its core was implemented in Unity3D, using a female 3D virtual assistant, named Vitória.

Results: Two COM-B components were identified as main targets for behavior change—psychological capability and reflective motivation; these were linked with four IFs—education, persuasion, enablement, and environmental restructuring. Eleven BCTs were, in turn, linked with the IFs. An example of a BCT is “problem solving”; it requires users to pinpoint factors influencing non-adherence and subsequently offers strategies to achieve the desired behavior. BCTs were operationalized into the dialogues with Vitória and into supplementary software features. Vitória communicates with users verbally and non-verbally, expressing emotions. Input options consist of buttons or recording values, such as medication taken.

Conclusion: The present approach enabled us to derive the most appropriate BCTs for our intervention. The use of an explicit bundle of BCTs, often overlooked in interventions promoting medication adherence, is expected to maximize effectiveness and facilitates replication. The first prototype is being refined with users and health professionals’ contributions. Future work includes subjecting the prototype to usability tests and a feasibility trial.

Introducing a Virtual Assistant to the Lab: A Voice User Interface for the Intuitive Control of Laboratory Instruments

The introduction of smart virtual assistants (VAs) and corresponding smart devices brought a new degree of freedom to our everyday lives. Voice-controlled and Internet-connected devices allow intuitive device controlling and monitoring from all around the globe and define a new era of human-machine interaction. Although VAs are especially successful in home automation, they also show great potential as artificial intelligence-driven laboratory assistants. Possible applications include stepwise reading of standard operating procedures (SOPs) and recipes, recitation of chemical substance or reaction parameters to a control, and readout of laboratory devices and sensors. In this study, we present a retrofitting approach to make standard laboratory instruments part of the Internet of Things (IoT). We established a voice user interface (VUI) for controlling those devices and reading out specific device data. A benchmark of the established infrastructure showed a high mean accuracy (95% ± 3.62) of speech command recognition and reveals high potential for future applications of a VUI within the laboratory. Our approach shows the general applicability of commercially available VAs as laboratory assistants and might be of special interest to researchers with physical impairments or low vision. The developed solution enables a hands-free device control, which is a crucial advantage within the daily laboratory routine.

Qualitative study of affective identities in dementia patients for the design of cognitive assistive technologies

Our overall aim is to develop an emotionally intelligent cognitive assistant (ICA) to help older adults with Alzheimer's disease (AD) to complete activities of daily living more independently. For improved adoption, such a system should take into account how individuals feel about who they are. This paper investigates different affective identities found in older care home residents with AD, leading to a computational characterization of these aspects and, thus, tailored prompts to each specific individual's identity in a way that potentially ensures smoother and more effective uptake and response. We report on a set of qualitative interviews with 12 older adult care home residents and caregivers. The interview covered life domains (family, origin, occupation, etc.), and feelings related to the ICA. All interviews were transcribed and analyzed to extract a set of affective identities, coded according to the social–psychological principles of affect control theory (ACT). Preliminary results show that a set of identities can be extracted for each participant (e.g. father, husband). Furthermore, our results provide support for the proposition that, while identities grounded in memories fade as a person loses their memory, habitual aspects of identity that reflect the overall “persona” may persist longer, even without situational context.