Using Natural Language Processing and Sentiment Analysis to Augment Traditional User-Centered Design: Development and Usability Study

Engagement of Older Adults in the Design, Implementation, and Evaluation of Artificial Intelligence Systems for Aging: A Scoping Review

Integration of artificial intelligence (AI) in health and healthcare, especially for older adults, has significantly advanced healthcare delivery. AI technologies, with capabilities such as self-learning and pattern recognition, are employed to address social isolation and monitor older adults' daily activities. However, rapid AI development often fails to consider the heterogeneous needs of older populations, which could exacerbate an existing digital divide and inequality. This scoping review examines older adults' involvement in AI system design, implementation, and evaluation of AI systems in health and healthcare literature, emphasizing the necessity of their input for beneficial AI systems. We conducted a scoping review according to PRISMA-SCR. We reviewed 17 studies, finding that half of these studies (n = 8) engaged older adults during the design phase, a small number (n = 3) during the evaluation stage, and even fewer (n = 2) involved older adults in the implementation stage. Despite AI's growing role, design processes often overlook older adults' needs. Our findings emphasize the need for inclusive, participatory design approaches to address ethical and equity challenges, enhancing user engagement and relevance. We also highlight how these approaches address the needs of older adults and improve outcomes. Specifically, we integrated evidence showing the practical benefits of these approaches for better accessibility, usability, and engagement among older adults. Although AI has potential to improve healthcare delivery, these approaches must be part of broader efforts to ensure ethical, inclusive, and equitable AI practices, especially in gerontology.

Natural language processing in mixed-methods evaluation of a digital sleep-alcohol intervention for young adults

We used natural language processing (NLP) in convergent mixed methods to evaluate young adults' experiences with Call it a Night (CIAN), a digital personalized feedback and coaching sleep-alcohol intervention. Young adults with heavy drinking (N = 120) were randomized to CIAN or controls (A + SM: web-based advice + self-monitoring or A: advice; clinicaltrials.gov, 8/31/18, #NCT03658954). Most CIAN participants (72.0%) preferred coaching to control interventions. Control participants found advice more helpful than CIAN participants (X2 = 27.34, p < 0.001). Most participants were interested in sleep factors besides alcohol and appreciated increased awareness through monitoring. NLP corroborated generally positive sentiments (M = 15.07(10.54)) and added critical insight that sleep (40%), not alcohol use (12%), was a main participant motivator. All groups had high adherence, satisfaction, and feasibility. CIAN (Δ = 0.48, p = 0.008) and A + SM (Δ = 0.55, p < 0.001) had higher reported effectiveness than A (F(2, 115) = 8.45, p < 0.001). Digital sleep-alcohol interventions are acceptable, and improving sleep and wellness may be important motivations for young adults.

ChatGPT-3.5 System Usability Scale early assessment among Healthcare Workers: Horizons of adoption in medical practice

Artificial intelligence (AI) chatbots, such as ChatGPT, have widely invaded all domains of human life. They have the potential to transform healthcare future. However, their effective implementation hinges on healthcare workers' (HCWs) adoption and perceptions. This study aimed to evaluate HCWs usability of ChatGPT three months post-launch in Saudi Arabia using the System Usability Scale (SUS). A total of 194 HCWs participated in the survey. Forty-seven percent were satisfied with their usage, 57 % expressed moderate to high trust in its ability to generate medical decisions. 58 % expected ChatGPT would improve patients' outcomes, even though 84 % were optimistic of its potential to improve the future of healthcare practice. They expressed possible concerns like recommending harmful medical decisions and medicolegal implications. The overall mean SUS score was 64.52, equivalent to 50 % percentile rank, indicating high marginal acceptability of the system. The strongest positive predictors of high SUS scores were participants' belief in AI chatbot's benefits in medical research, self-rated familiarity with ChatGPT and self-rated computer skills proficiency. Participants' learnability and ease of use score correlated positively but weakly. On the other hand, medical students and interns had significantly high learnability scores compared to others, while ease of use scores correlated very strongly with participants' perception of positive impact of ChatGPT on the future of healthcare practice. Our findings highlight the HCWs' perceived marginal acceptance of ChatGPT at the current stage and their optimism of its potential in supporting them in future practice, especially in the research domain, in addition to humble ambition of its potential to improve patients' outcomes particularly in regard of medical decisions. On the other end, it underscores the need for ongoing efforts to build trust and address ethical and legal concerns of AI implications in healthcare. The study contributes to the growing body of literature on AI chatbots in healthcare, especially addressing its future improvement strategies and provides insights for policymakers and healthcare providers about the potential benefits and challenges of implementing them in their practice.

Leveraging Natural Language Processing to Evaluate Young Adults' User Experiences with a Digital Sleep Intervention for Alcohol Use

Evaluating user experiences with digital interventions is critical to increase uptake and adherence, but traditional methods have limitations. We incorporated natural language processing (NLP) with convergent mixed methods to evaluate a personalized feedback and coaching digital sleep intervention for alcohol risk reduction: 'Call it a Night' (CIAN; N = 120). In this randomized clinical trial with young adults with heavy drinking, control conditions were A + SM: web-based advice + active and passive monitoring; and A: advice + passive monitoring. Findings converged to show that the CIAN treatment condition group found feedback and coaching most helpful, whereas participants across conditions generally found advice helpful. Further, most participants across groups were interested in varied whole-health sleep-related factors besides alcohol use (e.g., physical activity), and many appreciated increased awareness through monitoring with digital tools. All groups had high adherence, satisfaction, and reported feasibility, but participants in CIAN and A + SM reported significantly higher effectiveness than those in A. NLP corroborated positive sentiments across groups and added critical insight that sleep, not alcohol use, was a main participant motivator. Digital sleep interventions are an acceptable, novel alcohol treatment strategy, and improving sleep and overall wellness may be important motivations for young adults. Further, NLP provides an efficient convergent method for evaluating experiences with digital interventions.

Quantitative and qualitative analysis of individual experiences post botulinum toxin injection - United Kingdom Survey

Introduction

In the United Kingdom (UK), complications that arise following the administration of Botulinum Toxin are reported to the Medicines and Health Regulatory Agency (MHRA) via the Yellow Card Reporting Scheme. Over the past decade, there has been a significant increase in the number of non-surgical aesthetic procedures. Concerns have been raised that the MHRA is not fully capturing complications in terms of volume and impact on patients.

Aim

This novel study explores the lived experiences of individuals who have experienced an adverse event following administration of Botulinum Toxin for aesthetic purposes. Using a combination of qualitative and quantitative methodologies, this analysis evaluates data relating to long-lasting physical, psychological, emotional, and financial sequelae of complications arising from cosmetic Botulinum Toxin injections in the UK.

Methods

A mixed method, qualitative and quantitative approach was adopted to gain comprehensive insights into patients' experiences. A focus group which comprised patient representatives, psychologists, and researchers reached a consensus on a 17-question survey which was disseminated via social media channels. Deductive thematic analysis was used to analyse coded themes. Furthermore, for secondary analysis, sentiment analysis was used computationally as an innovative approach to identify and categorise free text responses associated with sentiments using natural language processing (NLP).

Results

In the study, 655 responses were received, with 287 (44%) of respondents completing all questions. The mean age of respondents was 42.6 years old. 94.1% of respondents identified as female. In the sample, 79% of respondents reported an adverse event following their procedure, with the most common event being reported as 'anxiety'. Findings revealed that 69% of respondents reported long-lasting adverse effects. From the responses, 68.4% reported not having recovered physically, 63.5% of respondents stated that they had not recovered emotionally from complications, and 61.7% said that they have not recovered psychologically. In addition, 84% of respondents stated that they do not know who regulates the aesthetics industry. Furthermore, 92% of participants reported that their clinic or practitioner did not inform them about the Yellow Card Reporting Scheme. The sentiment analysis using the AFINN Lexicon yielded adjusted scores ranging from -3 to +2, with a mean value of -1.58.

Conclusion

This is the largest survey in the UK completed by patients who experienced an adverse outcome following the aesthetic administration of Botulinum Toxin. Our study highlights the extent of the challenges faced by patients who experience an adverse event from physical, emotional, psychological, and financial perspectives. The lack of awareness of MHRA reporting structures and the lack of regulation within the UK's cosmetic injectables sector represent a significant public health challenge.

Boosting Delirium Identification Accuracy With Sentiment-Based Natural Language Processing: Mixed Methods Study

BACKGROUND

Delirium is an acute neurocognitive disorder that affects up to half of older hospitalized medical patients and can lead to dementia, longer hospital stays, increased health costs, and death. Although delirium can be prevented and treated, it is difficult to identify and predict.

OBJECTIVE

This study aimed to improve machine learning models that retrospectively identify the presence of delirium during hospital stays (eg, to measure the effectiveness of delirium prevention interventions) by using the natural language processing (NLP) technique of sentiment analysis (in this case a feature that identifies sentiment toward, or away from, a delirium diagnosis).

METHODS

Using data from the General Medicine Inpatient Initiative, a Canadian hospital data and analytics network, a detailed manual review of medical records was conducted from nearly 4000 admissions at 6 Toronto area hospitals. Furthermore, 25.74% (994/3862) of the eligible hospital admissions were labeled as having delirium. Using the data set collected from this study, we developed machine learning models with, and without, the benefit of NLP methods applied to diagnostic imaging reports, and we asked the question "can NLP improve machine learning identification of delirium?"

RESULTS

Among the eligible 3862 hospital admissions, 994 (25.74%) admissions were labeled as having delirium. Identification and calibration of the models were satisfactory. The accuracy and area under the receiver operating characteristic curve of the main model with NLP in the independent testing data set were 0.807 and 0.930, respectively. The accuracy and area under the receiver operating characteristic curve of the main model without NLP in the independent testing data set were 0.811 and 0.869, respectively. Model performance was also found to be stable over the 5-year period used in the experiment, with identification for a likely future holdout test set being no worse than identification for retrospective holdout test sets.

CONCLUSIONS

Our machine learning model that included NLP (ie, sentiment analysis in medical image description text mining) produced valid identification of delirium with the sentiment analysis, providing significant additional benefit over the model without NLP.

Usability evaluation of mHealth apps for elderly individuals: a scoping review

BACKGROUND

Usability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.

METHODS

Electronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.

RESULTS

Ninety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.

CONCLUSION

In the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Applications of natural language processing in ophthalmology: present and future

Advances in technology, including novel ophthalmic imaging devices and adoption of the electronic health record (EHR), have resulted in significantly increased data available for both clinical use and research in ophthalmology. While artificial intelligence (AI) algorithms have the potential to utilize these data to transform clinical care, current applications of AI in ophthalmology have focused mostly on image-based deep learning. Unstructured free-text in the EHR represents a tremendous amount of underutilized data in big data analyses and predictive AI. Natural language processing (NLP) is a type of AI involved in processing human language that can be used to develop automated algorithms using these vast quantities of available text data. The purpose of this review was to introduce ophthalmologists to NLP by (1) reviewing current applications of NLP in ophthalmology and (2) exploring potential applications of NLP. We reviewed current literature published in Pubmed and Google Scholar for articles related to NLP and ophthalmology, and used ancestor search to expand our references. Overall, we found 19 published studies of NLP in ophthalmology. The majority of these publications (16) focused on extracting specific text such as visual acuity from free-text notes for the purposes of quantitative analysis. Other applications included: domain embedding, predictive modeling, and topic modeling. Future ophthalmic applications of NLP may also focus on developing search engines for data within free-text notes, cleaning notes, automated question-answering, and translating ophthalmology notes for other specialties or for patients, especially with a growing interest in open notes. As medicine becomes more data-oriented, NLP offers increasing opportunities to augment our ability to harness free-text data and drive innovations in healthcare delivery and treatment of ophthalmic conditions.

Integrating Natural Language Processing and Interpretive Thematic Analyses to Gain Human-Centered Design Insights on HIV Mobile Health: Proof-of-Concept Analysis

BACKGROUND

HIV mobile health (mHealth) interventions often incorporate interactive peer-to-peer features. The user-generated content (UGC) created by these features can offer valuable design insights by revealing what topics and life events are most salient for participants, which can serve as targets for subsequent interventions. However, unstructured, textual UGC can be difficult to analyze. Interpretive thematic analyses can preserve rich narratives and latent themes but are labor-intensive and therefore scale poorly. Natural language processing (NLP) methods scale more readily but often produce only coarse descriptive results. Recent calls to advance the field have emphasized the untapped potential of combined NLP and qualitative analyses toward advancing user attunement in next-generation mHealth.

OBJECTIVE

In this proof-of-concept analysis, we gain human-centered design insights by applying hybrid consecutive NLP-qualitative methods to UGC from an HIV mHealth forum.

METHODS

UGC was extracted from Thrive With Me, a web app intervention for men living with HIV that includes an unstructured peer-to-peer support forum. In Python, topics were modeled by latent Dirichlet allocation. Rule-based sentiment analysis scored interactions by emotional valence. Using a novel ranking standard, the experientially richest and most emotionally polarized segments of UGC were condensed and then analyzed thematically in Dedoose. Design insights were then distilled from these themes.

RESULTS

The refined topic model detected K=3 topics: A: disease coping; B: social adversities; C: salutations and check-ins. Strong intratopic themes included HIV medication adherence, survivorship, and relationship challenges. Negative UGC often involved strong negative reactions to external media events. Positive UGC often focused on gratitude for survival, well-being, and fellow users' support.

CONCLUSIONS

With routinization, hybrid NLP-qualitative methods may be viable to rapidly characterize UGC in mHealth environments. Design principles point toward opportunities to align mHealth intervention features with the organically occurring uses captured in these analyses, for example, by foregrounding inspiring personal narratives and expressions of gratitude, or de-emphasizing anger-inducing media.

Promoting Research, Awareness, and Discussion on AI in Medicine Using #MedTwitterAI: A Longitudinal Twitter Hashtag Analysis

Background

Artificial intelligence (AI) has the potential to reshape medical practice and the delivery of healthcare. Online discussions surrounding AI's utility in these domains are increasingly emerging, likely due to considerable interest from healthcare practitioners, medical technology developers, and other relevant stakeholders. However, many practitioners and medical students report limited understanding and familiarity with AI.

Objective

To promote research, events, and resources at the intersection of AI and medicine for the online medical community, we created a Twitter-based campaign using the hashtag #MedTwitterAI.

Methods

In the present study, we analyze the use of #MedTwitterAI by tracking tweets containing this hashtag posted from 26th March, 2019 to 26th March, 2021, using the Symplur Signals hashtag analytics tool. The full text of all #MedTwitterAI tweets was also extracted and subjected to a natural language processing analysis.

Results

Over this time period, we identified 7,441 tweets containing #MedTwitterAI, posted by 1,519 unique Twitter users which generated 59,455,569 impressions. The most common identifiable locations for users including this hashtag in tweets were the United States (378/1,519), the United Kingdom (80/1,519), Canada (65/1,519), India (46/1,519), Spain (29/1,519), France (24/1,519), Italy (16/1,519), Australia (16/1,519), Germany (16/1,519), and Brazil (15/1,519). Tweets were frequently enhanced with links (80.2%), mentions of other accounts (93.9%), and photos (56.6%). The five most abundant single words were AI (artificial intelligence), patients, medicine, data, and learning. Sentiment analysis revealed an overall majority of positive single word sentiments (e.g., intelligence, improve) with 230 positive and 172 negative sentiments with a total of 658 and 342 mentions of all positive and negative sentiments, respectively. Most frequently mentioned negative sentiments were cancer, risk, and bias. Most common bigrams identified by Markov chain depiction were related to analytical methods (e.g., label-free detection) and medical conditions/biological processes (e.g., rare circulating tumor cells).

Conclusion

These results demonstrate the generated considerable interest of using #MedTwitterAI for promoting relevant content and engaging a broad and geographically diverse audience. The use of hashtags in Twitter-based campaigns can be an effective tool to raise awareness of interdisciplinary fields and enable knowledge-sharing on a global scale.

Human-Centered Design of Mobile Health Apps for Older Adults: Systematic Review and Narrative Synthesis

BACKGROUND

The world is aging. The number of older patients is on the rise, and along with it comes the burden of noncommunicable diseases, both clinical and economic. Attempts with mobile health (mHealth) have been made to remedy the situation with promising outcomes. Researchers have adopted human-centered design (HCD) in mHealth creation to ensure those promises become a reality.

OBJECTIVE

This systematic review aims to explore existing literature on relevant primary research and case studies to (1) illustrate how HCD can be used to create mHealth solutions for older adults and (2) summarize the overall process with recommendations specific to the older population.

METHODS

We conducted a systematic review to address the study objectives. IEEE Xplore, Medline via Ovid, PubMed, and Scopus were searched for HCD research of mHealth solutions for older adults. Two independent reviewers then included the papers if they (1) were written in English, (2) included participants equal to or older than 60 years old, (3) were primary research, and (4) reported about mHealth apps and their HCD developments from start to finish. The 2 reviewers continued to assess the included studies' qualities using the Mixed Methods Appraisal Tool (MMAT). A narrative synthesis was then carried out and completed.

RESULTS

Eight studies passed the eligibility criteria: 5 were mixed methods studies and 3 were case studies. Some studies were about the same mHealth projects with a total of 5 mHealth apps. The included studies differed in HCD goals, target groups, and details of their HCD methodologies. The HCD process was explored through narrative synthesis in 4 steps according to the International Standardization Organization (ISO) standard 9241-210: (1) understand and specify the context of use, (2) specify the user requirements, (3) produce design solutions to meet these requirements, and (4) evaluate the designs against requirements. The overall process and recommendations unique to older adults are summarized logically with structural order and time order based on the Minto pyramid principle and ISO 9241-210.

CONCLUSIONS

Findings show that HCD can be used to create mHealth solutions for older adults with positive outcomes. This review has also summarized practical HCD steps and additional suggestions based on existing literature in the subfield. However, evidence-based results are still limited because most included studies lacked details about their sampling methods and did not set objective and quantifiable goals, leading to failure to draw significant conclusions. More studies of HCD application on mHealth for older adults with measurable design goals and rigorous research strategy are warranted.

BandPass: A Bluetooth-Enabled Remote Monitoring Device for Sarcopenia

As the United States population ages, managing pathologies that largely affect older adults, including sarcopenia (i.e., loss of muscle mass and strength) represents a significant and growing clinical challenge. In addition to increased rates of sarcopenia with age, its incidence and impact increase after acute illness, increasing the risk of functional decline, institutionalization, or death. Resistance-based exercises promote muscle regeneration and strength and are an advised therapy for such patients. Yet, such therapeutic exercises are normally conducted either under direct clinical oversight or unsupervised by patients at home, where compliance rates are low. The presented device, BandPass, aims to create an integrated force data detection and acquisition system for monitoring and transmitting at-home exercise force data to patients and clinicians. A potentiometer-based sensor was integrated to a resistance exercise band through the use of custom designed electronics, which incorporated Bluetooth Low Energy (BLE) for wireless transmission to a mobile 'app'. A protocol for calibrating the device was developed using a range of loads and validated in static benchtop and dynamic testing. Data from a pilot study with 7 older adults was also collected and analyzed to test the device. BandPass is 94% accurate with a coefficient of variation (CoV) of 4.9% and sensitivity of 150g. The pilot study recorded 147 exercises, allowing for analysis on patients' exercise performances. BandPass was successfully able to measure force continuously over time during exercises, measure longitudinal compliance with exercises, and quantify force continuously over time. A mobile health (mHealth) force-sensing system allows for the remote monitoring of prescribed in-home resistance exercise band programs for at-risk older adults, bridging the gap between clinicians and patients.

Remote Rehabilitation: A Field-Based Feasibility Study of an mHealth Resistance Exercise Band

Sarcopenia is the age-related loss of muscle mass and strength that is associated with adverse health outcomes. Resistance-based exercises are effective for mitigation and enhancement of strength; however, adherence is low and challenging to measure when patients are at home. In a single-arm, pilot study of seven older adults, we conducted a field-based usability study evaluating the feasibility and acceptability of using a system consisting of a Bluetooth-connected resistance exercise band and tablet-based app which together we call BandPass in completing four different home-based exercises. The system measured a total of 147 exercises by participants with a mean duration of 94±66 seconds, completing an average of 30±20 repetitions. Though not all patients completed each exercise type, patients were positive about use: patient activation measure: 80.7±14; system usability scale: 6.9±2.9; and confidence in use: 7.7±2.7. The BandPass system demonstrated its ability to collect data on exercise type, force during an exercise, and duration of exercise when older adults use it for monitoring exercise at home.