Identification and Evaluation of Methodologies to Assess the Quality of Mobile Health Apps in High-, Low-, and Middle-Income Countries: Rapid Review

Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study

BACKGROUND

There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.

OBJECTIVE

The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).

METHODS

We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.

RESULTS

We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.

CONCLUSIONS

There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.

Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis

BACKGROUND

Technological advancements have opened the path for many technology providers to easily develop and introduce eHealth tools to the public. The use of these tools is increasingly recognized as a critical quality driver in health care; however, choosing a quality tool from the myriad of tools available for a specific health need does not come without challenges.

OBJECTIVE

This review aimed to systematically investigate the literature to understand the different approaches and criteria used to assess the quality and impact of eHealth tools by considering sociotechnical factors (from technical, social, and organizational perspectives).

METHODS

A structured search was completed following the participants, intervention, comparators, and outcomes framework. We searched the PubMed, Cochrane, Web of Science, Scopus, and ProQuest databases for studies published between January 2012 and January 2022 in English, which yielded 675 results, of which 40 (5.9%) studies met the inclusion criteria. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews of Interventions were followed to ensure a systematic process. Extracted data were analyzed using NVivo (QSR International), with a thematic analysis and narrative synthesis of emergent themes.

RESULTS

Similar measures from the different papers, frameworks, and initiatives were aggregated into 36 unique criteria grouped into 13 clusters. Using the sociotechnical approach, we classified the relevant criteria into technical, social, and organizational assessment criteria. Technical assessment criteria were grouped into 5 clusters: technical aspects, functionality, content, data management, and design. Social assessment criteria were grouped into 4 clusters: human centricity, health outcomes, visible popularity metrics, and social aspects. Organizational assessment criteria were grouped into 4 clusters: sustainability and scalability, health care organization, health care context, and developer.

CONCLUSIONS

This review builds on the growing body of research that investigates the criteria used to assess the quality and impact of eHealth tools and highlights the complexity and challenges facing these initiatives. It demonstrates that there is no single framework that is used uniformly to assess the quality and impact of eHealth tools. It also highlights the need for a more comprehensive approach that balances the social, organizational, and technical assessment criteria in a way that reflects the complexity and interdependence of the health care ecosystem and is aligned with the factors affecting users' adoption to ensure uptake and adherence in the long term.

Framework to assess the quality of mHealth apps: a mixed-method international case study protocol

INTRODUCTION

Healthcare professionals (HCPs) often recommend their patients to use a specific mHealth app as part of health promotion, disease prevention and patient self-management. There has been a significant growth in the number of HCPs downloading and using mobile health (mHealth) apps. Most mHealth apps that are available in app stores employ a 'star rating' system. This is based on user feedback on an app, but is highly subjective. Thus, the identification of quality mHealth apps which are deemed fit for purpose can be a difficult task for HCPs. Currently, there is no unified, validated standard guidelines for assessment of mHealth apps for patient safety, which can be used by HCPs. The Modified Enlight Suite (MES) is a quality assessment framework designed to provide a means for HCPs to evaluate mHealth apps before they are recommended to patients. MES was adapted from the original Enlight Suite for international use through a Delphi method, followed by preliminary validation process among a population predominantly consisting of medical students. This study aims to evaluate the applicability and validity of the MES, by HCPs, in low, middle and high income country settings.

METHODS AND ANALYSIS

MES will be evaluated through a mixed-method study, consisting of qualitative (focus group) and quantitative (survey instruments) research, in three target countries: Malaŵi (low income), South Africa (middle income) and Ireland (high income). The focus groups will be conducted through Microsoft Teams (Microsoft, Redmond, Washington, USA) and surveys will be conducted online using Qualtrics (Qualtrics International, Seattle, Washington, USA). Participants will be recruited through the help of national representatives in Malawi (Mzuzu University), South Africa (University of Fort Hare) and Ireland (University College Cork) by email invitation. Data analysis for the focus group will be by the means of thematic analysis. Data analysis for the survey will use descriptive statistics and use Cronbach alpha as an indicator of internal consistency of the MES. The construct validity of the mHealth app will be assessed by computing the confirmatory factor analysis using Amos.

ETHICS AND DISSEMINATION

The study has received ethical approval from the Social Research Ethics Committee (SREC) SREC/SOM/03092021/1 at University College Cork, Ireland, Malaŵi Research Ethics Committee (MREC), Malaŵi MZUNIREC/DOR/21/59 and Inter-Faculty Research Ethics Committee (IFREC) of University of Fort Hare (REC-2 70 710-028-RA). The results of the study will be disseminated through the internet, peer-reviewed journals and conference presentations.

Modification and Validation of an mHealth App Quality Assessment Methodology for International Use: Cross-sectional and eDelphi Studies

Background

Over 325,000 mobile health (mHealth) apps are available to download across various app stores. However, quality assurance in this field of medicine remains relatively undefined. Globally, around 84% of the population have access to mobile broadband networks. Given the potential for mHealth app use in health promotion and disease prevention, their role in patient care worldwide is ever apparent. Quality assurance regulations both nationally and internationally will take time to develop. Frameworks such as the Mobile App Rating Scale and Enlight Suite have demonstrated potential for use in the interim. However, these frameworks require adaptation to be suitable for international use.

Objective

This study aims to modify the Enlight Suite, a comprehensive app quality assessment methodology, to improve its applicability internationally and to assess the preliminary validity and reliability of this modified tool in practice.

Methods

A two-round Delphi study involving 7 international mHealth experts with varied backgrounds in health, technology, and clinical psychology was conducted to modify the Enlight Suite for international use and to improve its content validity. The Modified Enlight Suite (MES) was then used by 800 health care professionals and health care students in Ireland to assess a COVID-19 tracker app in an online survey. The reliability of the MES was assessed using Cronbach alpha, while the construct validity was evaluated using confirmatory factor analysis.

Results

The final version of the MES has 7 sections with 32 evaluating items. Of these items, 5 were novel and based on consensus for inclusion by Delphi panel members. The MES has satisfactory reliability with a Cronbach alpha score of .925. The subscales also demonstrated acceptable internal consistency. Similarly, the confirmatory factor analysis demonstrated a positive and significant factor loading for all 32 items in the MES with a modestly acceptable model fit, thus indicating the construct validity of the MES.

Conclusions

The Enlight Suite was modified to improve its international relevance to app quality assessment by introducing new items relating to cultural appropriateness, accessibility, and readability of mHealth app content. This study indicates both the reliability and validity of the MES for assessing the quality of mHealth apps in a high-income country, with further studies being planned to extrapolate these findings to low- and middle-income countries.

Toward a digital citizen lab for capturing data about alternative ways of self-managing chronic pain: An attitudinal user study

Background

Myriad psychosocial and cultural factors influence personal ways of coping with chronic pain (CP). Mobile health (mHealth) apps facilitate creation of citizen laboratories outside clinical frameworks. However, issues of safety, privacy and technostress must be addressed. This attitudinal user study aimed to assess whether persons with persistent pain (PwPP) would be open to sharing qualitative and quantitative data about their self-management of CP via mHealth platforms.

Methods

In March 2020, we invited PwPPs, their personal or medical caregivers, or those interested in the development of an app for researching alternative ways of self-managing CP to complete an anonymous survey. We formulated an attitudinal survey within the theoretical framework of stress to estimate whether the novelty, unpredictability, and risks of data-sharing via mHealth apps concerned users. Descriptive statistics (% Part/Group) were used to interpret the survey, and open comments were reflectively analyzed to identify emerging themes.

Results

Of 202 responses (June 2021), 127 identified as PwPPs (average age 43.86 ± 14.97; 100/127 female), and listed several primary and secondary CP diagnoses. In almost 90% of PwPPs, physical and emotional wellbeing were affected by CP. More than 90% of PwPPs used alternative therapies (acupuncture, homeopathy, massage therapy, etc.). Attitude toward mHealth apps were positive even though nearly half of PwPPs were unfamiliar with them. More than 72% of respondents were open to using a health-related app as a research tool for data collection in real life situations. Comprehensive data collection (especially about psychosocial factors) was the most important requirement. More respondents (especially medical professionals) were concerned about health hazards of misinformation communicated via health-related information and communication systems (maximum 80%) than about privacy (maximum 40%). Qualitative analyses revealed several promises and impediments to creation of data-sharing platforms for CP.

Conclusions

This study shows a general willingness among PwPPs to become partners in studying alternative pain management. Despite a generally positive attitude toward the concept of sharing complex personal data to advance research, heterogeneity of attitudes shaped by personal experiences must be considered. Our study underlines the need for any digital strategy for CP research to be person-centered and flexible.

Optimization of the Chronic Kidney Disease-Peritoneal Dialysis App to Improve Care for Patients on Peritoneal Dialysis in Northeast Thailand: User-Centered Design Study

Background

The prevalence of peritoneal dialysis (PD) in Thailand is increasing rapidly in part because of Thailand’s Peritoneal Dialysis First policy. PD is a home-based renal replacement therapy in which patients with chronic kidney disease perform up to 4 exchanges of dialysate fluid per day in the peritoneal cavity. Overhydration is one of the most common complications in patients on PD and is associated with increased morbidity and mortality. To monitor hydration status, patients collect hydration metrics, including body weight, blood pressure, urine output, and ultrafiltration volume, from each dialysis cycle and enter this information into a PD logbook. This information is reviewed bimonthly at PD clinic appointments. The chronic kidney disease-PD (CKD-PD) app with near-field communication (NFC) and optical character recognition (OCR) was developed to automate hydration metric collection. The information was displayed in the app for self-monitoring and uploaded to a database for real-time monitoring by the PD clinic staff. Early detection and treatment of overhydration could potentially reduce the morbidity and mortality related to overhydration.

Objective

This study aims to identify usability issues and technology adoption barriers for the CKD-PD app with NFC and OCR and a monitoring system and to use this information to make rapid cycle improvements.

Methods

A multidisciplinary team of nephrologists, PD clinic nurses, computer programmers, and engineers trained and observed 2 groups of 5 participants in the use of the CKD-PD app with NFC and OCR and a monitoring system. The participants were observed using technology in their homes in 3 phases. The data collected included the Unified Theory of Acceptance and Use of Technology questionnaire, think-aloud observation, user ratings, completion of hydration metrics, and upload of hydration metrics to the central database. These results were used by the team between phases to improve the functionality and usefulness of the app.

Results

The CKD-PD app with NFC and OCR and a monitoring system underwent 3 rapid improvement cycles. Issues were identified regarding the usability of the NFC and OCR data collection, app stability, user interface, hydration metric calculation, and display. NFC and OCR improved hydration metric capture; however, issues remained with their usability. App stability and user interface issues were corrected, and hydration metrics were successfully uploaded by the end of phase 3. Participants’ scores on technology adoption decreased but were still high, and there was enthusiasm for the self-monitoring and clinical communication features.

Conclusions

Our rapid cycle process improvement methodology identified and resolved key barriers and usability issues for the CKD-PD app with NFC and OCR and a monitoring system. We believe that this methodology can be accomplished with limited training in data collection, statistical analysis, and funding.

Mobile Health Apps Providing Information on Drugs for Adult Emergency Care: Systematic Search on App Stores and Content Analysis

Background

Drug-referencing apps are among the most frequently used by emergency health professionals. To date, no study has analyzed the quantity and quality of apps that provide information on emergency drugs.

Objective

This study aimed to identify apps designed to assist emergency professionals in managing drugs and to describe and analyze their characteristics.

Methods

We performed an observational, cross-sectional, descriptive study of apps that provide information on drugs for adult emergency care. The iOS and Android platforms were searched in February 2021. The apps were independently evaluated by 2 hospital clinical pharmacists. We analyzed developer affiliation, cost, updates, user ratings, and number of downloads. We also evaluated the main topic (emergency drugs or emergency medicine), the number of drugs described, the inclusion of bibliographic references, and the presence of the following drug information: commercial presentations, usual dosage, dose adjustment for renal failure, mechanism of action, therapeutic indications, contraindications, interactions with other medicinal products, use in pregnancy and breastfeeding, adverse reactions, method of preparation and administration, stability data, incompatibilities, identification of high-alert medications, positioning in treatment algorithms, information about medication reconciliation, and cost.

Results

Overall, 49 apps were identified. Of these 49 apps, 32 (65%) were found on both digital platforms; 11 (22%) were available only for Android, and 6 (12%) were available only for iOS. In total, 41% (20/49) of the apps required payment (ranging from €0.59 [US $0.64] to €179.99 [US $196.10]) and 22% (11/49) of the apps were developed by non–health care professionals. The mean weighted user rating was 4.023 of 5 (SD 0.71). Overall, 45% (22/49) of the apps focused on emergency drugs, and 55% (27/49) focused on emergency medicine. More than half (29/47, 62%) did not include bibliographic references or had not been updated for more than a year (29/49, 59%). The median number of drugs was 66 (range 4 to >5000). Contraindications (26/47, 55%) and adverse reactions (24/47, 51%) were found in only half of the apps. Less than half of the apps addressed dose adjustment for renal failure (15/47, 32%), interactions (10/47, 21%), and use during pregnancy and breastfeeding (15/47, 32%). Only 6% (3/47) identified high-alert medications, and 2% (1/47) included information about medication reconciliation. Health-related developer, main topic, and greater amount of drug information were not statistically associated with higher user ratings (P=.99, P=.09, and P=.31, respectively).

Conclusions

We provide a comprehensive review of apps with information on emergency drugs for adults. Information on authorship, drug characteristics, and bibliographic references is frequently scarce; therefore, we propose recommendations to consider when developing an app of these characteristics. Future efforts should be made to increase the regulation of drug-referencing apps and to conduct a more frequent and documented review of their clinical content.