Participatory Development and Pilot Testing of an Adolescent Health Promotion Chatbot

Current Landscape and Future Directions for Mental Health Conversational Agents for Youth: Scoping Review

BACKGROUND

Conversational agents (CAs; chatbots) are systems with the ability to interact with users using natural human dialogue. They are increasingly used to support interactive knowledge discovery of sensitive topics such as mental health topics. While much of the research on CAs for mental health has focused on adult populations, the insights from such research may not apply to CAs for youth.

OBJECTIVE

This study aimed to comprehensively evaluate the state-of-the-art research on mental health CAs for youth.

METHODS

Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we identified 39 peer-reviewed studies specific to mental health CAs designed for youth across 4 databases, including ProQuest, Scopus, Web of Science, and PubMed. We conducted a scoping review of the literature to evaluate the characteristics of research on mental health CAs designed for youth, the design and computational considerations of mental health CAs for youth, and the evaluation outcomes reported in the research on mental health CAs for youth.

RESULTS

We found that many mental health CAs (11/39, 28%) were designed as older peers to provide therapeutic or educational content to promote youth mental well-being. All CAs were designed based on expert knowledge, with a few that incorporated inputs from youth. The technical maturity of CAs was in its infancy, focusing on building prototypes with rule-based models to deliver prewritten content, with limited safety features to respond to imminent risk. Research findings suggest that while youth appreciate the 24/7 availability of friendly or empathetic conversation on sensitive topics with CAs, they found the content provided by CAs to be limited. Finally, we found that most (35/39, 90%) of the reviewed studies did not address the ethical aspects of mental health CAs, while youth were concerned about the privacy and confidentiality of their sensitive conversation data.

CONCLUSIONS

Our study highlights the need for researchers to continue to work together to align evidence-based research on mental health CAs for youth with lessons learned on how to best deliver these technologies to youth. Our review brings to light mental health CAs needing further development and evaluation. The new trend of large language model-based CAs can make such technologies more feasible. However, the privacy and safety of the systems should be prioritized. Although preliminary evidence shows positive trends in mental health CAs, long-term evaluative research with larger sample sizes and robust research designs is needed to validate their efficacy. More importantly, collaboration between youth and clinical experts is essential from the early design stages through to the final evaluation to develop safe, effective, and youth-centered mental health chatbots. Finally, best practices for risk mitigation and ethical development of CAs with and for youth are needed to promote their mental well-being.

A mixed methods exploratory study assessing youth knowledge, self-efficacy and mental health outcomes in transition to adult healthcare

Background

Transition from adolescence to adulthood is a change in status yielding self-sufficiency, independence, and participation in adult roles. There are numerous efforts to define transition elements and encourage formalized processes readying youth for adulthood, yet significant challenges remain.

Methods

We recruited individuals who were age 12 years or older and not transitioned or 18 years and older who transitioned within 24 months to complete questionnaires and participate in a focus group.

Results

25 participants (Mean age 17.9 years) completed questionnaires and 76 % participated in focus groups. Youth 18 years and older had mean scores in the reason for concern range on 7 of 9 NIH Toolbox scales de. Transition readiness was lower in adults across all five categories. Key themes include misunderstanding the concept of transition and self-determination in transition.

Conclusions

Few youth understood what "transition" entails. Most considered it a single issue, such as aging out of insurance or leaving their pediatric specialist. Youth self-efficacy and life outlook declines in the late teens and early adulthood, exacerbating perceived inability to self-manage and gain independence. Most concerning is the emotional well-being and self-esteem of our adults who scored in the reason for concern range in 7 of 9 scales whereas adolescent peers did not. A directed attempt to create self-efficacy for youth in transition through training in health management, understanding the meaning of transition and promoting a positive perception prior to transition is needed.

Recommendations for Mental Health Chatbot Conversations: An Integrative Review

AIM

To identify and synthesise recommendations and guidelines for mental health chatbot conversational design.

DESIGN

Integrative review.

METHODS

Suitable publications presenting recommendations or guidelines for mental health conversational design were included. The quality of included publications was assessed using Joanna Briggs Institute Critical Appraisal Tools. Thematic analysis was conducted.

DATA SOURCES

Primary searches limited to last 10 years were conducted in PubMed, Scopus, ACM Digital Library and EBSCO databases including APA PsycINFO, CINAHL, APA PsycArticles and MEDLINE in February 2023 and updated in October 2023. A secondary search was conducted in Google Scholar in May 2023.

RESULTS

Of 1684 articles screened, 16 publications were selected. Three overarching themes were developed: (1) explicit knowledge about chatbot design and domain, (2) knowing your audience and (3) creating a safe space to engage. Results highlight that creating pleasant and effective conversations with a mental health chatbot requires careful and professional planning in advance, defining the target group and working together with it to address its needs and preferences. It is essential to emphasise the pleasant user experience and safety from both technical and psychological perspectives.

CONCLUSION

Recommendations for mental health chatbot conversational design have evolved and become more specific in recent years. Recommendations set high standards for mental health chatbots. To meet that, co-design, explicit knowledge of the user needs, domain and conversational design is needed.

IMPLICATIONS FOR THE PROFESSION AND/OR PATIENT CARE

Mental health professionals participating in chatbot development can utilise this review. The results can also inform technical development teams not involving healthcare professionals directly.

IMPACT

Knowledge of developing mental health chatbot conversations appears scattered. In mental health chatbots, features that enhance the chatbot's ability to meet users' needs and increase safety should be considered. This review is useful for developers of mental health chatbots and other health applications used independently.

REPORTING METHOD

This integrative review was reported according to PRISMA guidelines, as applicable.

PATIENT OR PUBLIC CONTRIBUTION

No patient or public contribution.

A New Research Model for Artificial Intelligence-Based Well-Being Chatbot Engagement: Survey Study

BACKGROUND

Artificial intelligence (AI)-based chatbots have emerged as potential tools to assist individuals in reducing anxiety and supporting well-being.

OBJECTIVE

This study aimed to identify the factors that impact individuals' intention to engage and their engagement behavior with AI-based well-being chatbots by using a novel research model to enhance service levels, thereby improving user experience and mental health intervention effectiveness.

METHODS

We conducted a web-based questionnaire survey of adult users of well-being chatbots in China via social media. Our survey collected demographic data, as well as a range of measures to assess relevant theoretical factors. Finally, 256 valid responses were obtained. The newly applied model was validated through the partial least squares structural equation modeling approach.

RESULTS

The model explained 62.8% (R2) of the variance in intention to engage and 74% (R2) of the variance in engagement behavior. Affect (β=.201; P=.002), social factors (β=.184; P=.007), and compatibility (β=.149; P=.03) were statistically significant for the intention to engage. Habit (β=.154; P=.01), trust (β=.253; P<.001), and intention to engage (β=.464; P<.001) were statistically significant for engagement behavior.

CONCLUSIONS

The new extended model provides a theoretical basis for studying users' AI-based chatbot engagement behavior. This study highlights practical points for developers of AI-based well-being chatbots. It also highlights the importance of AI-based well-being chatbots to create an emotional connection with the users.

Perceptions and expectations of an artificially intelligent physical activity digital assistant - A focus group study

Artificially intelligent physical activity digital assistants that use the full spectrum of machine learning capabilities have not yet been developed and examined. This study aimed to explore potential users' perceptions and expectations of using such a digital assistant. Six 90-min online focus group meetings (n = 45 adults) were conducted. Meetings were recorded, transcribed and thematically analysed. Participants embraced the idea of a 'digital assistant' providing physical activity support. Participants indicated they would like to receive notifications from the digital assistant, but did not agree on the number, timing, tone and content of notifications. Likewise, they indicated that the digital assistant's personality and appearance should be customisable. Participants understood the need to provide information to the digital assistant to allow for personalisation, but varied greatly in the extent of information that they were willing to provide. Privacy issues aside, participants embraced the idea of using artificial intelligence or machine learning in return for a more functional and personal digital assistant. In sum, participants were ready for an artificially intelligent physical activity digital assistant but emphasised a need to personalise or customise nearly every feature of the application. This poses challenges in terms of cost and complexity of developing the application.

A necessary conversation to develop chatbots for HIV studies: qualitative findings from research staff, community advisory board members, and study participants

Chatbots increase business productivity by handling customer conversations instead of human agents. Similar rationale applies to use chatbots in the healthcare sector, especially for health coaches who converse with clients. Chatbots are nascent in healthcare. Study findings have been mixed in terms of engagement and their impact on outcomes. Questions remain as to chatbot acceptability with coaches and other providers; studies have focused on clients.To clarify perceived benefits of chatbots in HIV interventions we conducted virtual focus groups with 13 research staff, eight community advisory board members, and seven young adults who were HIV intervention trial participants (clients). Our HIV healthcare context is important. Clients represent a promising age demographic for chatbot uptake. They are a marginalized population warranting consideration to avoid technology that limits healthcare access.Focus group participants expressed the value of chatbots for HIV research staff and clients. Staff discussed how chatbot functions, such as automated appointment scheduling and service referrals, could reduce workloads while clients discussed the after-hours convenience of these functions. Participants also emphasized that chatbots should provide relatable conversation, reliable functionality, and would not be appropriate for all clients. Our findings underscore the need to further examine appropriate chatbot functionality in HIV interventions.

Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review

OBJECTIVES

Conversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions.

MATERIALS AND METHODS

We conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO).

RESULTS

The review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages.

DISCUSSION AND CONCLUSION

This review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research.

PROTOCOL REGISTRATION

The Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.

A mobile healthy lifestyle intervention to promote mental health in adolescence: a mixed-methods evaluation

BACKGROUND

A healthy lifestyle may improve mental health. It is yet not known whether and how a mobile intervention can be of help in achieving this in adolescents. This study investigated the effectiveness and perceived underlying mechanisms of the mobile health (mHealth) intervention #LIFEGOALS to promote healthy lifestyles and mental health. #LIFEGOALS is an evidence-based app with activity tracker, including self-regulation techniques, gamification elements, a support chatbot, and health narrative videos.

METHODS

A quasi-randomized controlled trial (N = 279) with 12-week intervention period and process evaluation interviews (n = 13) took place during the COVID-19 pandemic. Adolescents (12-15y) from the general population were allocated at school-level to the intervention (n = 184) or to a no-intervention group (n = 95). Health-related quality of life (HRQoL), psychological well-being, mood, self-perception, peer support, resilience, depressed feelings, sleep quality and breakfast frequency were assessed via a web-based survey; physical activity, sedentary time, and sleep routine via Axivity accelerometers. Multilevel generalized linear models were fitted to investigate intervention effects and moderation by pandemic-related measures. Interviews were coded using thematic analysis.

RESULTS

Non-usage attrition was high: 18% of the participants in the intervention group never used the app. An additional 30% stopped usage by the second week. Beneficial intervention effects were found for physical activity (χ21 = 4.36, P = .04), sedentary behavior (χ21 = 6.44, P = .01), sleep quality (χ21 = 6.11, P = .01), and mood (χ21 = 2.30, P = .02). However, effects on activity-related behavior were only present for adolescents having normal sports access, and effects on mood only for adolescents with full in-school education. HRQoL (χ22 = 14.72, P < .001), mood (χ21 = 6.03, P = .01), and peer support (χ21 = 13.69, P < .001) worsened in adolescents with pandemic-induced remote-education. Interviewees reported that the reward system, self-regulation guidance, and increased health awareness had contributed to their behavior change. They also pointed to the importance of social factors, quality of technology and autonomy for mHealth effectiveness.

CONCLUSIONS

#LIFEGOALS showed mixed results on health behaviors and mental health. The findings highlight the role of contextual factors for mHealth promotion in adolescence, and provide suggestions to optimize support by a chatbot and narrative episodes.

TRIAL REGISTRATION

ClinicalTrials.gov [NCT04719858], registered on 22/01/2021.

Development of a chatbot for school violence prevention among elementary school students in South Korea: a methodological study

PURPOSE

This study develops a chatbot for school violence prevention (C-SVP) among elementary school students.

METHODS

Among the analysis, design, development, implementation, and evaluation (ADDIE) models, ADD phases were applied to develop a C-SVP. Students' learning needs were identified by constructing content with a design that attracted their attention. Subsequently, a formative evaluation was conducted on the developed C-SVP to test its applicability by ten elementary school students targeting the 5th and 6th grades.

RESULTS

The chatbot was designed using KakaoTalk and named "School Guardian Angel." The formative evaluation revealed that the developed C-SVP was easily accessible and useful for elementary school students.

CONCLUSION

The developed C-SVP is expected to be effective in preventing violence among elementary school students. However, further research involving children of various age groups is required.

Examining Patient Engagement in Chatbot Development Approaches for Healthy Lifestyle and Mental Wellness Interventions: Scoping Review

BACKGROUND

Chatbots are growing in popularity as they offer a range of potential benefits to end users and service providers.

OBJECTIVE

Our scoping review aimed to explore studies that used 2-way chatbots to support healthy eating, physical activity, and mental wellness interventions. Our objectives were to report the nontechnical (eg, unrelated to software development) approaches for chatbot development and to examine the level of patient engagement in these reported approaches.

METHODS

Our team conducted a scoping review following the framework proposed by Arksey and O'Malley. Nine electronic databases were searched in July 2022. Studies were selected based on our inclusion and exclusion criteria. Data were then extracted and patient involvement was assessed.

RESULTS

16 studies were included in this review. We report several approaches to chatbot development, assess patient involvement where possible, and reveal the limited detail available on reporting of patient involvement in the chatbot implementation process. The reported approaches for development included: collaboration with knowledge experts, co-design workshops, patient interviews, prototype testing, the Wizard of Oz (WoZ) procedure, and literature review. Reporting of patient involvement in development was limited; only 3 of the 16 included studies contained sufficient information to evaluate patient engagement using the Guidance for Reporting Involvement of Patients and Public (GRIPP2).

CONCLUSIONS

The approaches reported in this review and the identified limitations can guide the inclusion of patient engagement and the improved documentation of engagement in the chatbot development process for future health care research. Given the importance of end user involvement in chatbot development, we hope that future research will more systematically report on chatbot development and more consistently and actively engage patients in the codevelopment process.

Feasibility and Acceptability of Chatbots for Nutrition and Physical Activity Health Promotion Among Adolescents: Systematic Scoping Review With Adolescent Consultation

BACKGROUND

Reducing lifestyle risk behaviors among adolescents depends on access to age-appropriate health promotion information. Chatbots-computer programs designed to simulate conversations with human users-have the potential to deliver health information to adolescents to improve their lifestyle behaviors and support behavior change, but research on the feasibility and acceptability of chatbots in the adolescent population is unknown.

OBJECTIVE

This systematic scoping review aims to evaluate the feasibility and acceptability of chatbots in nutrition and physical activity interventions among adolescents. A secondary aim is to consult adolescents to identify features of chatbots that are acceptable and feasible.

METHODS

We searched 6 electronic databases from March to April 2022 (MEDLINE, Embase, Joanna Briggs Institute, the Cumulative Index to Nursing and Allied Health, the Association for Computing Machinery library, and the IT database Institute of Electrical and Electronics Engineers). Peer-reviewed studies were included that were conducted in the adolescent population (10-19 years old) without any chronic disease, except obesity or type 2 diabetes, and assessed chatbots used nutrition or physical activity interventions or both that encouraged individuals to meet dietary or physical activity guidelines and support positive behavior change. Studies were screened by 2 independent reviewers, with any queries resolved by a third reviewer. Data were extracted into tables and collated in a narrative summary. Gray literature searches were also undertaken. Results of the scoping review were presented to a diverse youth advisory group (N=16, 13-18 years old) to gain insights into this topic beyond what is published in the literature.

RESULTS

The search identified 5558 papers, with 5 (0.1%) studies describing 5 chatbots meeting the inclusion criteria. The 5 chatbots were supported by mobile apps using a combination of the following features: personalized feedback, conversational agents, gamification, and monitoring of behavior change. Of the 5 studies, 2 (40.0%) studies focused on nutrition, 2 (40.0%) studies focused on physical activity, and 1 (20.0%) focused on both nutrition and physical activity. Feasibility and acceptability varied across the 5 studies, with usage rates above 50% in 3 (60.0%) studies. In addition, 3 (60.0%) studies reported health-related outcomes, with only 1 (20.0%) study showing promising effects of the intervention. Adolescents presented novel concerns around the use of chatbots in nutrition and physical activity interventions, including ethical concerns and the use of false or misleading information.

CONCLUSIONS

Limited research is available on chatbots in adolescent nutrition and physical activity interventions, finding insufficient evidence on the acceptability and feasibility of chatbots in the adolescent population. Similarly, adolescent consultation identified issues in the design features that have not been mentioned in the published literature. Therefore, chatbot codesign with adolescents may help ensure that such technology is feasible and acceptable to an adolescent population.

Developing and Testing a Chatbot to Integrate HIV Education Into Family Planning Clinic Waiting Areas in Lusaka, Zambia

BACKGROUND

To maximize protection against both unintended pregnancy and HIV, it is important that family planning (FP) services integrate HIV counseling, both to support method choice and identify potential HIV services of interest, such as pre-exposure prophylaxis (PrEP). However, FP providers often lack sufficient time and knowledge to address HIV vulnerability with clients. To potentially offload some of the initial HIV counseling burden from FP providers, we developed and tested a chatbot that provided information about HIV and dual protection to FP clients in waiting areas of FP clinics in Lusaka, Zambia.

CHATBOT DEVELOPMENT

We drafted a scripted conversation and tested it in English in formative workshops with Zambian women between the ages of 15 and 49 years. After translating the content to Bemba and Nyanja, we conducted a second round of workshops to validate the translations, before uploading the content into the chatbot platform.

CHATBOT USER TEST

Thirty volunteers tested the chatbot in 3 Lusaka FP clinics, completing an exit survey to provide feedback. A large majority (83%) said they learned new HIV information from the chatbot. Twenty (67%) learned about PrEP for the first time through the chat. Most (96%) reported discussing HIV with the provider, after engaging with the chatbot. In response to an open-ended question, several testers volunteered that they wanted to learn more about PrEP.

CONCLUSIONS

Pre-consultation waiting-area time is an underutilized opportunity to impart HIV information to FP clients, thereby preparing them to discuss their dual HIV and pregnancy prevention needs when they see their providers. FP clients expressed particular interest in learning more about PrEP, underscoring the importance of integrating HIV into FP services.

How Far Can Conversational Agents Contribute to IBD Patient Health Care—A Review of the Literature

Modern societies are facing health and healthcare challenges as never seen before. The digital world in which we are living today considers digital health interventions such as “internet-delivered” therapy (e-Therapy) or mobile apps as an integrated part of healthcare systems. Digital transformation in health care requires the active involvement of patients as the central part of healthcare interventions. In the case of chronic health conditions, such as inflammatory bowel disease (IBD), it is believed that the adoption of new digital tools helps to maintain and extend the health and care of patients, optimizing the course of the treatment of the disease. The study goal was to undertake a literature review associating the use of chatbot technology with IBD patients' health care. This study intends to support digital product developments, mainly chatbot for IBD or other chronic diseases. The work was carried out through two literature review phases. The first one was based on a systematic approach and the second was a scoping review focused only on Frontiers Journals. This review followed a planned protocol for search and selection strategy that was created by a research team discussion. Chatbot technology for chronic disease self-management can have high acceptance and usability levels. The more interaction with a chatbot, the more patients are able to increase their self-care practice, but there is a challenge. The chatbot ontology to personalize the communication still needed to have strong guidelines helping other researchers to define which Electronic Medical Records (EMRs) should be used in the chatbots to improve the user satisfaction, engagement, and dialog quality. The literature review showed us both evidence and success of these tools in other health disorders. Some of them revealed a huge potential for conversational agents as a part of digital health interventions.

Nonusage Attrition of Adolescents in an mHealth Promotion Intervention and the Role of Socioeconomic Status: Secondary Analysis of a 2-Arm Cluster-Controlled Trial

Background

Mobile health (mHealth) interventions may help adolescents adopt healthy lifestyles. However, attrition in these interventions is high. Overall, there is a lack of research on nonusage attrition in adolescents, particularly regarding the role of socioeconomic status (SES).

Objective

The aim of this study was to focus on the role of SES in the following three research questions (RQs): When do adolescents stop using an mHealth intervention (RQ1)? Why do they report nonusage attrition (RQ2)? Which intervention components (ie, self-regulation component, narrative, and chatbot) prevent nonusage attrition among adolescents (RQ3)?

Methods

A total of 186 Flemish adolescents (aged 12-15 years) participated in a 12-week mHealth program. Log data were monitored to measure nonusage attrition and usage duration for the 3 intervention components. A web-based questionnaire was administered to assess reasons for attrition. A survival analysis was conducted to estimate the time to attrition and determine whether this differed according to SES (RQ1). Descriptive statistics were performed to map the attrition reasons, and Fisher exact tests were used to determine if these reasons differed depending on the educational track (RQ2). Mixed effects Cox proportional hazard regression models were used to estimate the associations between the use duration of the 3 components during the first week and attrition. An interaction term was added to the regression models to determine whether associations differed by the educational track (RQ3).

Results

After 12 weeks, 95.7% (178/186) of the participants stopped using the app. 30.1% (56/186) of the adolescents only opened the app on the installation day, and 44.1% (82/186) stopped using the app in the first week. Attrition at any given time during the intervention period was higher for adolescents from the nonacademic educational track compared with those from the academic track. The other SES indicators (family affluence and perceived financial situation) did not explain attrition. The most common reasons for nonusage attrition among participants were perceiving that the app did not lead to behavior change, not liking the app, thinking that they already had a sufficiently healthy lifestyle, using other apps, and not being motivated by the environment. Attrition reasons did not differ depending on the educational track. More time spent in the self-regulation and narrative components during the first week was associated with lower attrition, whereas chatbot use duration was not associated with attrition rates. No moderating effects of SES were observed in the latter association.

Conclusions

Nonusage attrition was high, especially among adolescents in the nonacademic educational track. The reported reasons for attrition were diverse, with no statistical differences according to the educational level. The duration of the use of the self-regulation and narrative components during the first week may prevent attrition for both educational tracks.

Trial Registration

ClinicalTrials.gov NCT04719858; http://clinicaltrials.gov/ct2/show/NCT04719858