Identification and Evaluation of Methodologies to Assess the Quality of Mobile Health Apps in High-, Low-, and Middle-Income Countries: Rapid Review

Persian adaptation and validation of the user version of The Mobile Application Rating Scale (uMARS)

OBJECTIVE

Health applications (apps) enable patients to make decisions regarding their health utilizing digitally-enabled processes for care, expand accessibility to healthcare service delivery, and raise public awareness of health. In many languages, there is still a lack of an instrument that assesses how patients perceive apps. This study aims to adapt and validate the User Version of the Mobile Application Rating Scale (uMARS) in Persian and evaluate the overall quality of the Persian meditation apps.

METHODS

This cross-sectional study population comprises 86 healthcare workers in a health center in the west of Iran. The sample size was determined using Cochran's formula. First, the uMARS was translated into Persian. Then, validity was assessed using the Content Validity Ratio (CVR), Content Validity Index (CVI), and face validity. Cronbach's alpha and Intraclass Correlation Coefficients (ICC) were used to assess reliability. Among the 124 meditation apps, Aramia met the criteria included in the study and was selected to evaluate objective quality, subjective quality, and perceived impact.

RESULTS

The majority of the participants were female (82.55%). More than half of the participants had a bachelor's degree (58.14%). The CVR, CVI, ICC, and Cronbach's alpha values were obtained as 0.79. 0.90, 0.93, and 0.86, respectively. These Findings revealed that the Persian version of the questionnaire has sufficient reliability and validity. Among the three subscales, perceived impact received the highest mean score (3.96 ± 0.37), and a total score of 3.74 ± 1.04 was obtained.

CONCLUSIONS

A Persian version of uMARS is a valid and reliable tool for evaluating the quality of mobile health apps from the user's point of view during the development and testing process and improving the app's quality.

Psychometric evaluation of the System Usability Scale in the context of a childrearing app co-designed for low- and middle-income countries

Objective

The System Usability Scale (SUS) demonstrates good psychometric properties for a range of technologies; however, its reliability and factor structure in the context of a childrearing application (app) and variation across cultures remains largely unexplored. This study investigates the reliability and factor structure of the SUS in the context of a childrearing app that was co-designed for and implemented in diverse low- and middle-income countries (LMICs).

Methods

Parents and caregivers of children aged 0-5 years in five LMICs completed the SUS after having access to the app for a minimum of 2 and maximum of 24 weeks. Survey data from participants (n = 668) was analysed using exploratory and confirmatory factor analysis methods.

Results

The bi-factor model shows the best fit to data (comparative fit index = 0.998; Tucker Lewis Index = 0.996; standardised root mean square residual = 0.033). Further analysis suggests that usability and learnability subscales provide additional information not contained in the total SUS score. A two-sample t-test shows that younger caregivers, employed full- or part-time, and with fewer children reported significantly better app usability.

Conclusion

The SUS has good psychometric properties, and it is a valid and reliable tool for assessing the usability of mobile apps when used by parents and other caregivers for children's socioemotional and cognitive development. However, it is not essentially unidimensional and appears to have a multidimensional structure that could be specific to our context owing to variations in users' experience, culture, and language. The findings have implications for other mobile health interventions implemented in contexts with cultural and linguistic differences.

Quality Assessment of Digital Health Apps: Umbrella Review

BACKGROUND

With an increasing number of digital health apps available in app stores, it is important to assess these technologies reliably regarding their quality. This is done to mitigate the risks associated with their use. There are many different guidelines, methods, and metrics available to assess digital health apps with regard to their quality.

OBJECTIVE

This study aimed to give a holistic summary of the current methods and "condition agnostic" frameworks that are broadly applicable for the quality assessment of all digital health apps.

METHODS

A systematic search of literature was conducted on 4 databases: Scopus, PubMed, ACM Digital Library, and IEEE Xplore. We followed the PICOS (Population, Patient, or Problem; Intervention; Comparison; Outcomes; and Study Design) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodologies when conducting this umbrella review. The search was conducted on January 26, 2024, for review articles published between 2018 and 2023. We identified 4781 candidate papers for inclusion; after title and abstract screening, 39 remained. After full-text analysis, we included 15 review articles in the full review.

RESULTS

Of the 15 review articles, scoping reviews were the most common (n=6, 40%), followed by systematic reviews (n=4, 27%), narrative reviews (n=4, 27%), and a rapid review (n=1, 7%). A total of 4 (27%) review articles proposed assessment criteria for digital health apps. "Data privacy and/or security" was the most mentioned criterion (n=13, 87%) and "Cost" was the least mentioned criterion (n=1, 7%) for the assessment of digital health apps. The Mobile App Rating Scale was the most frequently used framework for quality assessment of digital health apps.

CONCLUSIONS

There is a lack of unity or consolidation across identified frameworks, as most do not meet all the identified criteria from the reviewed articles. Safety concerns associated with the use of digital health apps may be mitigated with the use of quality frameworks.

Challenges and Alternatives to Evaluation Methods and Regulation Approaches for Medical Apps as Mobile Medical Devices: International and Multidisciplinary Focus Group Discussion

BACKGROUND

The rapid proliferation of medical apps has transformed the health care landscape by giving patients and health care providers unprecedented access to personalized health information and services. However, concerns regarding the effectiveness and safety of medical apps have raised questions regarding the efficacy of randomized controlled trials (RCTs) in the evaluation of such apps and as a requirement for their regulation as mobile medical devices.

OBJECTIVE

This study aims to address this issue by investigating alternative methods, apart from RCTs, for evaluating and regulating medical apps.

METHODS

Using a qualitative approach, a focus group study with 46 international and multidisciplinary public health experts was conducted at the 17th World Congress on Public Health in May 2023 in Rome, Italy. The group was split into 3 subgroups to gather in-depth insights into alternative approaches for evaluating and regulating medical apps. We conducted a policy analysis on the current regulation of medical apps as mobile medical devices for the 4 most represented countries in the workshop: Italy, Germany, Canada, and Australia. We developed a logic model that combines the evaluation and regulation domains on the basis of these findings.

RESULTS

The focus group discussions explored the strengths and limitations of the current evaluation and regulation methods and identified potential alternatives that could enhance the quality and safety of medical apps. Although RCTs were only explicitly mentioned in the German regulatory system as one of many options, an analysis of chosen evaluation methods for German apps on prescription pointed toward a "scientific reflex" where RCTs are always the chosen evaluation method. However, this method has substantial limitations when used to evaluate digital interventions such as medical apps. Comparable results were observed during the focus group discussions, where participants expressed similar experiences with their own evaluation approaches. In addition, the participants highlighted numerous alternatives to RCTs. These alternatives can be used at different points during the life cycle of a digital intervention to assess its efficacy and potential harm to users.

CONCLUSIONS

It is crucial to recognize that unlike analog tools, digital interventions constantly evolve, posing challenges to inflexible evaluation methods such as RCTs. Potential risks include high dropout rates, decreased adherence, and nonsignificant results. However, existing regulations do not explicitly advocate for other evaluation methodologies. Our research highlighted the necessity of overcoming the gap between regulatory demands to demonstrate safety and efficacy of medical apps and evolving scientific practices, ensuring that digital health innovation is evaluated and regulated in a way that considers the unique characteristics of mobile medical devices.

Clinical Validation of Digital Healthcare Solutions: State of the Art, Challenges and Opportunities

Digital health technologies (DHTs) at the intersection of health, medical informatics, and business aim to enhance patient care through personalised digital approaches. Ensuring the efficacy and reliability of these innovations demands rigorous clinical validation. A PubMed literature review (January 2006 to July 2023) identified 1250 papers, highlighting growing academic interest. A focused narrative review (January 2018 to July 2023) delved into challenges, highlighting issues such as diverse regulatory landscapes, adoption issues in complex healthcare systems, and a plethora of evaluation frameworks lacking pragmatic guidance. Existing frameworks often omit crucial criteria, neglect empirical evidence, and clinical effectiveness is rarely included as a criterion for DHT quality. The paper underscores the urgency of addressing challenges in accreditation, adoption, business models, and integration to safeguard the quality, efficacy, and safety of DHTs. A pivotal illustration of collaborative efforts to address these challenges is exemplified by the Digital Health Validation Center, dedicated to generating clinical evidence of innovative healthcare technologies and facilitating seamless technology transfer. In conclusion, it is necessary to harmonise evaluation approaches and frameworks, improve regulatory clarity, and commit to collaboration to integrate rigorous clinical validation and empirical evidence throughout the DHT life cycle.

Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study

BACKGROUND

There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.

OBJECTIVE

The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).

METHODS

We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.

RESULTS

We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.

CONCLUSIONS

There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.

Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis

BACKGROUND

Technological advancements have opened the path for many technology providers to easily develop and introduce eHealth tools to the public. The use of these tools is increasingly recognized as a critical quality driver in health care; however, choosing a quality tool from the myriad of tools available for a specific health need does not come without challenges.

OBJECTIVE

This review aimed to systematically investigate the literature to understand the different approaches and criteria used to assess the quality and impact of eHealth tools by considering sociotechnical factors (from technical, social, and organizational perspectives).

METHODS

A structured search was completed following the participants, intervention, comparators, and outcomes framework. We searched the PubMed, Cochrane, Web of Science, Scopus, and ProQuest databases for studies published between January 2012 and January 2022 in English, which yielded 675 results, of which 40 (5.9%) studies met the inclusion criteria. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews of Interventions were followed to ensure a systematic process. Extracted data were analyzed using NVivo (QSR International), with a thematic analysis and narrative synthesis of emergent themes.

RESULTS

Similar measures from the different papers, frameworks, and initiatives were aggregated into 36 unique criteria grouped into 13 clusters. Using the sociotechnical approach, we classified the relevant criteria into technical, social, and organizational assessment criteria. Technical assessment criteria were grouped into 5 clusters: technical aspects, functionality, content, data management, and design. Social assessment criteria were grouped into 4 clusters: human centricity, health outcomes, visible popularity metrics, and social aspects. Organizational assessment criteria were grouped into 4 clusters: sustainability and scalability, health care organization, health care context, and developer.

CONCLUSIONS

This review builds on the growing body of research that investigates the criteria used to assess the quality and impact of eHealth tools and highlights the complexity and challenges facing these initiatives. It demonstrates that there is no single framework that is used uniformly to assess the quality and impact of eHealth tools. It also highlights the need for a more comprehensive approach that balances the social, organizational, and technical assessment criteria in a way that reflects the complexity and interdependence of the health care ecosystem and is aligned with the factors affecting users' adoption to ensure uptake and adherence in the long term.

Framework to assess the quality of mHealth apps: a mixed-method international case study protocol

INTRODUCTION

Healthcare professionals (HCPs) often recommend their patients to use a specific mHealth app as part of health promotion, disease prevention and patient self-management. There has been a significant growth in the number of HCPs downloading and using mobile health (mHealth) apps. Most mHealth apps that are available in app stores employ a 'star rating' system. This is based on user feedback on an app, but is highly subjective. Thus, the identification of quality mHealth apps which are deemed fit for purpose can be a difficult task for HCPs. Currently, there is no unified, validated standard guidelines for assessment of mHealth apps for patient safety, which can be used by HCPs. The Modified Enlight Suite (MES) is a quality assessment framework designed to provide a means for HCPs to evaluate mHealth apps before they are recommended to patients. MES was adapted from the original Enlight Suite for international use through a Delphi method, followed by preliminary validation process among a population predominantly consisting of medical students. This study aims to evaluate the applicability and validity of the MES, by HCPs, in low, middle and high income country settings.

METHODS AND ANALYSIS

MES will be evaluated through a mixed-method study, consisting of qualitative (focus group) and quantitative (survey instruments) research, in three target countries: Malaŵi (low income), South Africa (middle income) and Ireland (high income). The focus groups will be conducted through Microsoft Teams (Microsoft, Redmond, Washington, USA) and surveys will be conducted online using Qualtrics (Qualtrics International, Seattle, Washington, USA). Participants will be recruited through the help of national representatives in Malawi (Mzuzu University), South Africa (University of Fort Hare) and Ireland (University College Cork) by email invitation. Data analysis for the focus group will be by the means of thematic analysis. Data analysis for the survey will use descriptive statistics and use Cronbach alpha as an indicator of internal consistency of the MES. The construct validity of the mHealth app will be assessed by computing the confirmatory factor analysis using Amos.

ETHICS AND DISSEMINATION

The study has received ethical approval from the Social Research Ethics Committee (SREC) SREC/SOM/03092021/1 at University College Cork, Ireland, Malaŵi Research Ethics Committee (MREC), Malaŵi MZUNIREC/DOR/21/59 and Inter-Faculty Research Ethics Committee (IFREC) of University of Fort Hare (REC-2 70 710-028-RA). The results of the study will be disseminated through the internet, peer-reviewed journals and conference presentations.

Modification and Validation of an mHealth App Quality Assessment Methodology for International Use: Cross-sectional and eDelphi Studies

Background

Over 325,000 mobile health (mHealth) apps are available to download across various app stores. However, quality assurance in this field of medicine remains relatively undefined. Globally, around 84% of the population have access to mobile broadband networks. Given the potential for mHealth app use in health promotion and disease prevention, their role in patient care worldwide is ever apparent. Quality assurance regulations both nationally and internationally will take time to develop. Frameworks such as the Mobile App Rating Scale and Enlight Suite have demonstrated potential for use in the interim. However, these frameworks require adaptation to be suitable for international use.

Objective

This study aims to modify the Enlight Suite, a comprehensive app quality assessment methodology, to improve its applicability internationally and to assess the preliminary validity and reliability of this modified tool in practice.

Methods

A two-round Delphi study involving 7 international mHealth experts with varied backgrounds in health, technology, and clinical psychology was conducted to modify the Enlight Suite for international use and to improve its content validity. The Modified Enlight Suite (MES) was then used by 800 health care professionals and health care students in Ireland to assess a COVID-19 tracker app in an online survey. The reliability of the MES was assessed using Cronbach alpha, while the construct validity was evaluated using confirmatory factor analysis.

Results

The final version of the MES has 7 sections with 32 evaluating items. Of these items, 5 were novel and based on consensus for inclusion by Delphi panel members. The MES has satisfactory reliability with a Cronbach alpha score of .925. The subscales also demonstrated acceptable internal consistency. Similarly, the confirmatory factor analysis demonstrated a positive and significant factor loading for all 32 items in the MES with a modestly acceptable model fit, thus indicating the construct validity of the MES.

Conclusions

The Enlight Suite was modified to improve its international relevance to app quality assessment by introducing new items relating to cultural appropriateness, accessibility, and readability of mHealth app content. This study indicates both the reliability and validity of the MES for assessing the quality of mHealth apps in a high-income country, with further studies being planned to extrapolate these findings to low- and middle-income countries.

Toward a digital citizen lab for capturing data about alternative ways of self-managing chronic pain: An attitudinal user study

Background

Myriad psychosocial and cultural factors influence personal ways of coping with chronic pain (CP). Mobile health (mHealth) apps facilitate creation of citizen laboratories outside clinical frameworks. However, issues of safety, privacy and technostress must be addressed. This attitudinal user study aimed to assess whether persons with persistent pain (PwPP) would be open to sharing qualitative and quantitative data about their self-management of CP via mHealth platforms.

Methods

In March 2020, we invited PwPPs, their personal or medical caregivers, or those interested in the development of an app for researching alternative ways of self-managing CP to complete an anonymous survey. We formulated an attitudinal survey within the theoretical framework of stress to estimate whether the novelty, unpredictability, and risks of data-sharing via mHealth apps concerned users. Descriptive statistics (% Part/Group) were used to interpret the survey, and open comments were reflectively analyzed to identify emerging themes.

Results

Of 202 responses (June 2021), 127 identified as PwPPs (average age 43.86 ± 14.97; 100/127 female), and listed several primary and secondary CP diagnoses. In almost 90% of PwPPs, physical and emotional wellbeing were affected by CP. More than 90% of PwPPs used alternative therapies (acupuncture, homeopathy, massage therapy, etc.). Attitude toward mHealth apps were positive even though nearly half of PwPPs were unfamiliar with them. More than 72% of respondents were open to using a health-related app as a research tool for data collection in real life situations. Comprehensive data collection (especially about psychosocial factors) was the most important requirement. More respondents (especially medical professionals) were concerned about health hazards of misinformation communicated via health-related information and communication systems (maximum 80%) than about privacy (maximum 40%). Qualitative analyses revealed several promises and impediments to creation of data-sharing platforms for CP.

Conclusions

This study shows a general willingness among PwPPs to become partners in studying alternative pain management. Despite a generally positive attitude toward the concept of sharing complex personal data to advance research, heterogeneity of attitudes shaped by personal experiences must be considered. Our study underlines the need for any digital strategy for CP research to be person-centered and flexible.

Optimization of the Chronic Kidney Disease-Peritoneal Dialysis App to Improve Care for Patients on Peritoneal Dialysis in Northeast Thailand: User-Centered Design Study

Background

The prevalence of peritoneal dialysis (PD) in Thailand is increasing rapidly in part because of Thailand’s Peritoneal Dialysis First policy. PD is a home-based renal replacement therapy in which patients with chronic kidney disease perform up to 4 exchanges of dialysate fluid per day in the peritoneal cavity. Overhydration is one of the most common complications in patients on PD and is associated with increased morbidity and mortality. To monitor hydration status, patients collect hydration metrics, including body weight, blood pressure, urine output, and ultrafiltration volume, from each dialysis cycle and enter this information into a PD logbook. This information is reviewed bimonthly at PD clinic appointments. The chronic kidney disease-PD (CKD-PD) app with near-field communication (NFC) and optical character recognition (OCR) was developed to automate hydration metric collection. The information was displayed in the app for self-monitoring and uploaded to a database for real-time monitoring by the PD clinic staff. Early detection and treatment of overhydration could potentially reduce the morbidity and mortality related to overhydration.

Objective

This study aims to identify usability issues and technology adoption barriers for the CKD-PD app with NFC and OCR and a monitoring system and to use this information to make rapid cycle improvements.

Methods

A multidisciplinary team of nephrologists, PD clinic nurses, computer programmers, and engineers trained and observed 2 groups of 5 participants in the use of the CKD-PD app with NFC and OCR and a monitoring system. The participants were observed using technology in their homes in 3 phases. The data collected included the Unified Theory of Acceptance and Use of Technology questionnaire, think-aloud observation, user ratings, completion of hydration metrics, and upload of hydration metrics to the central database. These results were used by the team between phases to improve the functionality and usefulness of the app.

Results

The CKD-PD app with NFC and OCR and a monitoring system underwent 3 rapid improvement cycles. Issues were identified regarding the usability of the NFC and OCR data collection, app stability, user interface, hydration metric calculation, and display. NFC and OCR improved hydration metric capture; however, issues remained with their usability. App stability and user interface issues were corrected, and hydration metrics were successfully uploaded by the end of phase 3. Participants’ scores on technology adoption decreased but were still high, and there was enthusiasm for the self-monitoring and clinical communication features.

Conclusions

Our rapid cycle process improvement methodology identified and resolved key barriers and usability issues for the CKD-PD app with NFC and OCR and a monitoring system. We believe that this methodology can be accomplished with limited training in data collection, statistical analysis, and funding.

Mobile Health Apps Providing Information on Drugs for Adult Emergency Care: Systematic Search on App Stores and Content Analysis

Background

Drug-referencing apps are among the most frequently used by emergency health professionals. To date, no study has analyzed the quantity and quality of apps that provide information on emergency drugs.

Objective

This study aimed to identify apps designed to assist emergency professionals in managing drugs and to describe and analyze their characteristics.

Methods

We performed an observational, cross-sectional, descriptive study of apps that provide information on drugs for adult emergency care. The iOS and Android platforms were searched in February 2021. The apps were independently evaluated by 2 hospital clinical pharmacists. We analyzed developer affiliation, cost, updates, user ratings, and number of downloads. We also evaluated the main topic (emergency drugs or emergency medicine), the number of drugs described, the inclusion of bibliographic references, and the presence of the following drug information: commercial presentations, usual dosage, dose adjustment for renal failure, mechanism of action, therapeutic indications, contraindications, interactions with other medicinal products, use in pregnancy and breastfeeding, adverse reactions, method of preparation and administration, stability data, incompatibilities, identification of high-alert medications, positioning in treatment algorithms, information about medication reconciliation, and cost.

Results

Overall, 49 apps were identified. Of these 49 apps, 32 (65%) were found on both digital platforms; 11 (22%) were available only for Android, and 6 (12%) were available only for iOS. In total, 41% (20/49) of the apps required payment (ranging from €0.59 [US $0.64] to €179.99 [US $196.10]) and 22% (11/49) of the apps were developed by non–health care professionals. The mean weighted user rating was 4.023 of 5 (SD 0.71). Overall, 45% (22/49) of the apps focused on emergency drugs, and 55% (27/49) focused on emergency medicine. More than half (29/47, 62%) did not include bibliographic references or had not been updated for more than a year (29/49, 59%). The median number of drugs was 66 (range 4 to >5000). Contraindications (26/47, 55%) and adverse reactions (24/47, 51%) were found in only half of the apps. Less than half of the apps addressed dose adjustment for renal failure (15/47, 32%), interactions (10/47, 21%), and use during pregnancy and breastfeeding (15/47, 32%). Only 6% (3/47) identified high-alert medications, and 2% (1/47) included information about medication reconciliation. Health-related developer, main topic, and greater amount of drug information were not statistically associated with higher user ratings (P=.99, P=.09, and P=.31, respectively).

Conclusions

We provide a comprehensive review of apps with information on emergency drugs for adults. Information on authorship, drug characteristics, and bibliographic references is frequently scarce; therefore, we propose recommendations to consider when developing an app of these characteristics. Future efforts should be made to increase the regulation of drug-referencing apps and to conduct a more frequent and documented review of their clinical content.