Engagement, Satisfaction, and Mental Health Outcomes Across Different Residential Subgroup Users of a Digital Mental Health Relational Agent: Exploratory Single-Arm Study

BACKGROUND

Mental illness is a pervasive worldwide public health issue. Residentially vulnerable populations, such as those living in rural medically underserved areas (MUAs) or mental health provider shortage areas (MHPSAs), face unique access barriers to mental health care. Despite the growth of digital mental health interventions using relational agent technology, little is known about their use patterns, efficacy, and favorability among residentially vulnerable populations.

OBJECTIVE

This study aimed to explore differences in app use, therapeutic alliance, mental health outcomes, and satisfaction across residential subgroups (metropolitan, nonmetropolitan, or rural), MUAs (yes or no), and MHPSAs (yes or no) among users of a smartphone-based, digital mental health intervention, Woebot LIFE (WB-LIFE). WB-LIFE was designed to help users better understand and manage their moods and features a relational agent, Woebot, that converses through text-based messages.

METHODS

We used an exploratory study that examined data from 255 adults enrolled in an 8-week, single-arm trial of WB-LIFE. Analyses compared levels of app use and therapeutic alliance total scores as well as subscales (goal, task, and bond), mental health outcomes (depressive and anxiety symptoms, stress, resilience, and burnout), and program satisfaction across residential subgroups.

RESULTS

Few study participants resided in nonmetropolitan (25/255, 10%) or rural (3/255, 1%) areas, precluding estimates across this variable. Despite a largely metropolitan sample, nearly 39% (99/255) resided in an MUA and 55% (141/255) in an MHPSA. There were no significant differences in app use or satisfaction by MUA or MHPSA status. There also were no differences in depressive symptoms, anxiety, stress, resilience, or burnout, with the exception of MUA participants having higher baseline depressive symptoms among those starting in the moderate range or higher (Patient Health Questionnaire-8 item scale≥10) than non-MUA participants (mean 16.50 vs 14.41, respectively; P=.01). Although working alliance scores did not differ by MHPSA status, those who resided in an MUA had higher goal (2-tailed t203.47=2.21; P=.03), and bond (t203.47=1.94; P=.05) scores at day 3 (t192.98=2.15; P=.03), and higher goal scores at week 8 (t186.19=2.28; P=.02) as compared with those not living in an MUA.

CONCLUSIONS

Despite the study not recruiting many participants from rural or nonmetropolitan populations, sizable proportions resided in an MUA or an MHPSA. Analyses revealed few differences in app use, therapeutic alliance, mental health outcomes (including baseline levels), or satisfaction across MUA or MHPSA status over the 8-week study. Findings suggest that vulnerable residential populations may benefit from using digital agent-guided cognitive behavioral therapy.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05672745; https://clinicaltrials.gov/study/NCT05672745.

A Relational Agent Intervention for Adolescents Seeking Mental Health Treatment: Protocol for a Randomized Controlled Trial

BACKGROUND

Unmet pediatric mental health (MH) needs are growing as rates of pediatric depression and anxiety dramatically increase. Access to care is limited by multiple factors, including a shortage of clinicians trained in developmentally specific, evidence-based services. Novel approaches to MH care delivery, including technology-leveraged and readily accessible options, need to be evaluated in service of expanding evidence-based services to youths and their families. Preliminary evidence supports the use of Woebot, a relational agent that digitally delivers guided cognitive behavioral therapy (CBT) through a mobile app, for adults with MH concerns. However, no studies have evaluated the feasibility and acceptability of such app-delivered relational agents specifically for adolescents with depression and/or anxiety within an outpatient MH clinic, nor compared them to other MH support services.

OBJECTIVE

This paper describes the protocol for a randomized controlled trial evaluating the feasibility and acceptability of an investigational device, Woebot for Adolescents (W-GenZD), within an outpatient MH clinic for youths presenting with depression and/or anxiety. The study's secondary aim will compare the clinical outcomes of self-reported depressive symptoms with W-GenZD and a telehealth-delivered CBT-based skills group (CBT-group). Tertiary aims will evaluate additional clinical outcomes and therapeutic alliance between adolescents in W-GenZD and the CBT-group.

METHODS

Participants include youths aged 13-17 years with depression and/or anxiety seeking care from an outpatient MH clinic at a children's hospital. Eligible youths will have no recent safety concerns or complex comorbid clinical diagnoses; have no concurrent individual therapy; and, if on medications, are on stable doses, based on clinical screening and as well as study-specific criteria.

RESULTS

Recruitment began in May 2022. As of December 8, 2022, we have randomized 133 participants.

CONCLUSIONS

Establishing the feasibility and acceptability of W-GenZD within an outpatient MH clinical setting will add to the field's current understanding of the utility and implementation considerations of this MH care service modality. The study will also evaluate the noninferiority of W-GenZD against the CBT-group. Findings may also have implications for patients, families, and providers looking for additional MH support options for adolescents seeking help for their depression and/or anxiety. Such options expand the menu of supports for youths with lower-intensity needs as well as possibly reduce waitlists and optimize clinician deployment toward more severe cases.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05372913; https://clinicaltrials.gov/ct2/show/NCT05372913.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/44940.

Conversational Agents for Health and Well-being Across the Life Course: Protocol for an Evidence Map

BACKGROUND

Conversational agents, which we defined as computer programs that are designed to simulate two-way human conversation by using language and are potentially supplemented with nonlanguage modalities, offer promising avenues for health interventions for different populations across the life course. There is a lack of open-access and user-friendly resources for identifying research trends and gaps and pinpointing expertise across international centers.

OBJECTIVE

Our aim is to provide an overview of all relevant evidence on conversational agents for health and well-being across the life course. Specifically, our objectives are to identify, categorize, and synthesize-through visual formats and a searchable database-primary studies and reviews in this research field.

METHODS

An evidence map was selected as the type of literature review to be conducted, as it optimally corresponded to our aim. We systematically searched 8 databases (MEDLINE; CINAHL; Web of Science; Scopus; the Cochrane, ACM, IEEE, and Joanna Briggs Institute databases; and Google Scholar). We will perform backward citation searching on all included studies. The first stage of a double-stage screening procedure, which was based on abstracts and titles only, was conducted by using predetermined eligibility criteria for primary studies and reviews. An operational screening procedure was developed for streamlined and consistent screening across the team. Double data extraction will be performed with previously piloted data collection forms. We will appraise systematic reviews by using A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2. Primary studies and reviews will be assessed separately in the analysis. Data will be synthesized through descriptive statistics, bivariate statistics, and subgroup analysis (if appropriate) and through high-level maps such as scatter and bubble charts. The development of the searchable database will be informed by the research questions and data extraction forms.

RESULTS

As of April 2021, the literature search in the eight databases was concluded, yielding a total of 16,351 records. The first stage of screening, which was based on abstracts and titles only, resulted in the selection of 1282 records of primary studies and 151 records of reviews. These will be subjected to second-stage screening. A glossary with operational definitions for supporting the study selection and data extraction stages was drafted. The anticipated completion date is October 2021.

CONCLUSIONS

Our wider definition of a conversational agent and the broad scope of our evidence map will explicate trends and gaps in this field of research. Additionally, our evidence map and searchable database of studies will help researchers to avoid fragmented research efforts and wasteful redundancies. Finally, as part of the Harnessing the Power of Conversational e-Coaches for Health and Well-being Through Swiss-Portuguese Collaboration project, our work will also inform the development of an international taxonomy on conversational agents for health and well-being, thereby contributing to terminology standardization and categorization.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/26680.

Safety and Acceptability of a Natural Language Artificial Intelligence Assistant to Deliver Clinical Follow-up to Cataract Surgery Patients: Proposal

Background

Due to an aging population, the demand for many services is exceeding the capacity of the clinical workforce. As a result, staff are facing a crisis of burnout from being pressured to deliver high-volume workloads, driving increasing costs for providers. Artificial intelligence (AI), in the form of conversational agents, presents a possible opportunity to enable efficiency in the delivery of care.

Objective

This study aims to evaluate the effectiveness, usability, and acceptability of Dora agent: Ufonia’s autonomous voice conversational agent, an AI-enabled autonomous telemedicine call for the detection of postoperative cataract surgery patients who require further assessment. The objectives of this study are to establish Dora’s efficacy in comparison with an expert clinician, determine baseline sensitivity and specificity for the detection of true complications, evaluate patient acceptability, collect evidence for cost-effectiveness, and capture data to support further development and evaluation.

Methods

Using an implementation science construct, the interdisciplinary study will be a mixed methods phase 1 pilot establishing interobserver reliability of the system, usability, and acceptability. This will be done using the following scales and frameworks: the system usability scale; assessment of Health Information Technology Interventions in Evidence-Based Medicine Evaluation Framework; the telehealth usability questionnaire; and the Non-Adoption, Abandonment, and Challenges to the Scale-up, Spread and Suitability framework.

Results

The evaluation is expected to show that conversational technology can be used to conduct an accurate assessment and that it is acceptable to different populations with different backgrounds. In addition, the results will demonstrate how successfully the system can be delivered in organizations with different clinical pathways and how it can be integrated with their existing platforms.

Conclusions

The project’s key contributions will be evidence of the effectiveness of AI voice conversational agents and their associated usability and acceptability.

International Registered Report Identifier (IRRID)

PRR1-10.2196/27227

Agents of change: Understanding the therapeutic processes associated with the helpfulness of therapy for mental health problems with relational agent MYLO

Objective

To understand the therapeutic processes associated with the helpfulness of an online relational agent intervention, Manage Your Life Online (MYLO).

Methods

Fifteen participants experiencing a mental health related problem used Manage Your Life Online for 2 weeks. At follow-up, the participants each identified two helpful and two unhelpful questions posed by Manage Your Life Online within a single intervention session. Qualitative interviews were conducted and analyzed using thematic and content analysis to gain insight into the process of therapy with Manage Your Life Online.

Results

MYLO appeared acceptable to participants with a range of presenting problems. Questions enabling free expression, increased awareness, and new insights were key to a helpful intervention. The findings were consistent with the core processes of therapeutic change, according to Perceptual Control Theory, a unifying theory of psychological distress. Questions that elicited intense emotions, were repetitive, confusing, or inappropriate were identified as unhelpful and were associated with disengagement or loss of faith in Manage Your Life Online.

Conclusions

The findings provide insight into the likely core therapy processes experienced as helpful or hindering and outlines further ways to optimize acceptability of Manage Your Life Online.

A Protocol-Driven, Bedside Digital Conversational Agent to Support Nurse Teams and Mitigate Risks of Hospitalization in Older Adults: Case Control Pre-Post Study

BACKGROUND

Hospitalized older adults often experience isolation and disorientation while receiving care, placing them at risk for many inpatient complications, including loneliness, depression, delirium, and falls. Embodied conversational agents (ECAs) are technological entities that can interact with people through spoken conversation. Some ECAs are also relational agents, which build and maintain socioemotional relationships with people across multiple interactions. This study utilized a novel form of relational ECA, provided by Care Coach (care.coach, inc): an animated animal avatar on a tablet device, monitored and controlled by live health advocates. The ECA implemented algorithm-based clinical protocols for hospitalized older adults, such as reorienting patients to mitigate delirium risk, eliciting toileting needs to prevent falls, and engaging patients in social interaction to facilitate social engagement. Previous pilot studies of the Care Coach avatar have demonstrated the ECA's usability and efficacy in home-dwelling older adults. Further study among hospitalized older adults in a larger experimental trial is needed to demonstrate its effectiveness.

OBJECTIVE

The aim of the study was to examine the effect of a human-in-the-loop, protocol-driven relational ECA on loneliness, depression, delirium, and falls among diverse hospitalized older adults.

METHODS

This was a clinical trial of 95 adults over the age of 65 years, hospitalized at an inner-city community hospital. Intervention participants received an avatar for the duration of their hospital stay; participants on a control unit received a daily 15-min visit from a nursing student. Measures of loneliness (3-item University of California, Los Angeles Loneliness Scale), depression (15-item Geriatric Depression Scale), and delirium (confusion assessment method) were administered upon study enrollment and before discharge.

RESULTS

Participants who received the avatar during hospitalization had lower frequency of delirium at discharge (P<.001), reported fewer symptoms of loneliness (P=.01), and experienced fewer falls than control participants. There were no significant differences in self-reported depressive symptoms.

CONCLUSIONS

The study findings validate the use of human-in-the-loop, relational ECAs among diverse hospitalized older adults.

Using Relational Agents to Promote Family Communication Around Type 1 Diabetes Self-Management in the Diabetes Family Teamwork Online Intervention: Longitudinal Pilot Study

Background

Family conflict can reduce adolescent adherence to type 1 diabetes management tasks. The Family Teamwork in-person intervention was shown to be efficacious in reducing conflict and low adherence to diabetes-related tasks. Its reach and potential impact, however, were limited by the need to deliver the intervention sessions in person. Relational agents (ie, computerized versions of humans) have been shown to appeal to diverse audiences and may be an acceptable replacement for a human in technology-based behavior change interventions.

Objective

The purpose of this paper is to present the results of a pilot study assessing feasibility and acceptability of Diabetes Family Teamwork Online, an adapted version of the Family Teamwork intervention, delivered over the internet and guided by a relational agent.

Methods

Parent-adolescent dyads were recruited through a diabetes care clinic at a large tertiary care hospital in the southwestern United States. A one-group design, with assessments at baseline, immediate postintervention, and 3 months later, was used to assess feasibility. A priori feasibility criteria included an assessment of recruitment, completion, attrition, program satisfaction, therapeutic alliance, attitudes toward the relational agent, and data collection. The institutional review board at Baylor College of Medicine approved the protocol (H-37245).

Results

Twenty-seven adolescents aged 10 to 15 years with type 1 diabetes and their parents were enrolled. Criteria used to assess feasibility were (1) recruitment goals were met (n=20), (2) families completed ≥75% of the modules, (3) attrition rate was ≤10%, (4) program satisfaction was high (≥80% of families), (5) therapeutic alliance was high (average score of ≥60/84), (6) families expressed positive attitudes toward the relational agent (average item score of ≥5 on ≥4 items), (7) ≥80% of data were collected at post 1 and post 2, and (8) few technical issues (≤10%) occurred during intervention delivery. All feasibility criteria were met. Qualitative data confirmed that adolescents and parents had positive reactions to both the content and approach.

Conclusions

The Diabetes Family Teamwork Online intervention proved to be a feasible and acceptable method for enhancing communication around diabetes management tasks in families with an adolescent who has type 1 diabetes.

International Registered Report Identifier (IRRID)

RR2-10.2196/resprot.5817

Development of a Complex Intervention to Improve Adherence to Antidiabetic Medication in Older People Using an Anthropomorphic Virtual Assistant Software

Introduction: Improving adherence to antidiabetic medication is crucial, resulting in improved health outcomes, cost reduction, and minimization of waste. A lack of underlying theory in existing interventions may explain the limited success in sustaining behavior change. This paper describes the development of a theory and evidence-based complex intervention to improve adherence to oral antidiabetics in older people via a software prototype with an anthropomorphic virtual assistant.

Methods: The Behavior Change Wheel (BCW) was used to develop a theoretical understanding of the change process, corresponding to the first phase of the Medical Research Council Framework for developing and evaluating complex interventions. At the BCW core is a model of human behavior (COM-B), which posits that human behavior (B) results from the interaction between capabilities (C), opportunities (O), and motivation (M). Literature-derived medication adherence determinants were mapped onto COM-B components. Then, intervention functions (IFs) were selected employing the APEASE criteria. Finally, standardized behavior change techniques (BCTs) were chosen based on their suitability and their effectiveness on medication adherence trials. The prototype was developed for android devices; its core was implemented in Unity3D, using a female 3D virtual assistant, named Vitória.

Results: Two COM-B components were identified as main targets for behavior change—psychological capability and reflective motivation; these were linked with four IFs—education, persuasion, enablement, and environmental restructuring. Eleven BCTs were, in turn, linked with the IFs. An example of a BCT is “problem solving”; it requires users to pinpoint factors influencing non-adherence and subsequently offers strategies to achieve the desired behavior. BCTs were operationalized into the dialogues with Vitória and into supplementary software features. Vitória communicates with users verbally and non-verbally, expressing emotions. Input options consist of buttons or recording values, such as medication taken.

Conclusion: The present approach enabled us to derive the most appropriate BCTs for our intervention. The use of an explicit bundle of BCTs, often overlooked in interventions promoting medication adherence, is expected to maximize effectiveness and facilitates replication. The first prototype is being refined with users and health professionals’ contributions. Future work includes subjecting the prototype to usability tests and a feasibility trial.

How should a virtual agent present psychoeducation? Influence of verbal and textual presentation on adherence

BACKGROUND AND OBJECTIVE

With the rise of autonomous e-mental health applications, virtual agents can play a major role in improving trustworthiness, therapy outcome and adherence. In these applications, it is important that patients adhere in the sense that they perform the tasks, but also that they adhere to the specific recommendations on how to do them well. One important construct in improving adherence is psychoeducation, information on the why and how of therapeutic interventions. In an e-mental health context, this can be delivered in two different ways: verbally by a (virtual) embodied conversational agent or just via text on the screen. The aim of this research is to study which presentation mode is preferable for improving adherence.

METHODS

This study takes the approach of evaluating a specific part of a therapy, namely psychoeducation. This was done in a non-clinical sample, to first test the general constructs of the human-computer interaction. We performed an experimental study on the effect of presentation mode of psychoeducation on adherence. In this study, we took into account the moderating effects of attitude towards the virtual agent and recollection of the information. Within the paradigm of expressive writing, we asked participants (n= 46) to pick one of their worst memories to describe in a digital diary after receiving verbal or textual psychoeducation.

RESULTS AND CONCLUSION

We found that both the attitude towards the virtual agent and how well the psychoeducation was recollected were positively related to adherence in the form of task execution. Moreover, after controlling for the attitude to the agent and recollection, presentation of psychoeducation via text resulted in higher adherence than verbal presentation by the virtual agent did.

Use of Relational Agents to Improve Family Communication in Type 1 Diabetes: Methods

Background

Physiological and environmental risk factors interact to undermine blood glucose control during early adolescence. This has been documented to be associated with family conflict and poor adherence to diabetes management tasks. Family Teamwork is an efficacious program demonstrated to enhance family communication and reduce conflict during this vulnerable period. It was designed to be delivered to families in-person, which limited reach and potential impact.

Objective

The purpose of this paper is to present the protocol for adapting Family Teamwork for Web-based delivery.

Methods

Formative research with health care providers, parents, and adolescents will help modify Family Teamwork for Web-based delivery by a relational agent (ie, a computerized character with human-like features and actions). Sessions will be interactive, requiring both parent and adolescent participation, with the relational agent serving as a health coach. After programming, usability testing will be conducted to help ensure the program is easy to use. Video and instructional materials will be developed to facilitate use, and a small pilot study will be conducted to assess feasibility. Families will provide written informed consent prior to participation in any phase of the study. The Institutional Review Board at Baylor College of Medicine reviewed and approved the protocol (H-37245).

Results

Formative research is underway. No results are available at this time.

Conclusions

This research has the potential to make an important contribution to diabetes management by using technology to enhance the reach of an efficacious program.

Improving Access to Online Health Information With Conversational Agents: A Randomized Controlled Experiment

BACKGROUND

Conventional Web-based search engines may be unusable by individuals with low health literacy for finding health-related information, thus precluding their use by this population.

OBJECTIVE

We describe a conversational search engine interface designed to allow individuals with low health and computer literacy identify and learn about clinical trials on the Internet.

METHODS

A randomized trial involving 89 participants compared the conversational search engine interface (n=43) to the existing conventional keyword- and facet-based search engine interface (n=46) for the National Cancer Institute Clinical Trials database. Each participant performed 2 tasks: finding a clinical trial for themselves and finding a trial that met prespecified criteria.

RESULTS

Results indicated that all participants were more satisfied with the conversational interface based on 7-point self-reported satisfaction ratings (task 1: mean 4.9, SD 1.8 vs mean 3.2, SD 1.8, P<.001; task 2: mean 4.8, SD 1.9 vs mean 3.2, SD 1.7, P<.001) compared to the conventional Web form-based interface. All participants also rated the trials they found as better meeting their search criteria, based on 7-point self-reported scales (task 1: mean 3.7, SD 1.6 vs mean 2.7, SD 1.8, P=.01; task 2: mean 4.8, SD 1.7 vs mean 3.4, SD 1.9, P<.01). Participants with low health literacy failed to find any trials that satisfied the prespecified criteria for task 2 using the conventional search engine interface, whereas 36% (5/14) were successful at this task using the conversational interface (P=.05).

CONCLUSIONS

Conversational agents can be used to improve accessibility to Web-based searches in general and clinical trials in particular, and can help decrease recruitment bias against disadvantaged populations.

Response to a Relational Agent by Hospital Patients with Depressive Symptoms

Depression affects approximately 15% of the US population, and is recognized as an important risk factor for poor outcomes among patients with various illnesses. Automated health education and behavior change programs have the potential to help address many of the shortcomings in health care. However, the role of these systems in the care of patients with depression has been insufficiently examined. In the current study, we sought to evaluate how hospitalized medical patients would respond to a computer animated conversational agent that has been developed to provide information in an empathic fashion about a patient's hospital discharge plan. In particular, we sought to examine how patients who have a high level of depressive symptoms respond to this system. Therapeutic alliance-the trust and belief that a patient and provider have in working together to achieve a desired therapeutic outcome- was used as the primary outcome measure, since it has been shown to be important in predicting outcomes across a wide range of health problems, including depression. In an evaluation of 139 hospital patients who interacted with the agent at the time of discharge, all patients, regardless of depressive symptoms, rated the agent very high on measures of satisfaction and ease of use, and most preferred receiving their discharge information from the agent compared to their doctors or nurses in the hospital. In addition, we found that patients with symptoms indicative of major depression rated the agent significantly higher on therapeutic alliance compared to patients who did not have major depressive symptoms. We conclude that empathic agents represent a promising technology for patient assessment, education and counseling for those most in need of comfort and caring in the inpatient setting.

Context Awareness in a Handheld Exercise Agent

Work towards the development of a handheld health counseling agent designed to promote physical activity is described. Previous work on automated health counselors is discussed, along with the affordances of mobility and context awareness for health behavior interventions. We present a general-purpose software architecture for the rapid design and deployment of mobile health counseling agents. We also describe the results of an initial field trial in which such a mobile agent plays the role of an exercise coach designed to motivate users to walk more. Results were mixed. We found that the context awareness mechanism that was implemented for detecting walking led to greater user-agent social bonding, but less walking in study participants.