Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps

Do pain management apps use evidence-based psychological components? A systematic review of app content and quality

Background

With hundreds of pain management apps on the Canadian marketplace, it can be challenging for patients and clinicians to select effective and evidence-based mobile health (mHealth) apps that address pain from a biopsychosocial perspective.

Aims

The aim of this study is to identify pain management apps within the Canadian app marketplaces to aid clinicians in recommending apps.

Methods

The iOS and Android marketplaces were systematically searched to identify pain management apps that included at least one core component of cognitive behavioral therapy (CBT) or mindfulness- and acceptance-based therapies. Selected apps were assessed using a researcher developed psychological components checklist, and the Mobile App Rating Scale (MARS). These two measures provided a robust assessment of the apps' technical abilities and psychological principles being implemented.

Results

Five hundred eight pain management apps were identified, yet only 12 included a psychological component and were available for evaluation. On average, apps contained 8.10 out of 18 psychological components (SD = 2.77) with a MARS quality rating of 4.02 out of 5 (SD = 0.32). The most common psychological components were grounded in CBT, including psychoeducation, sleep hygiene, behavioral activation, coping skills training, and social support. Among the least commonly included components were goal setting, values, and culture/diversity. Two-thirds of the apps involved health care practitioners in their development, but independent scientific review of apps was scarce.

Conclusion

The highest scoring apps (Curable, Pathways, Vivify) are highlighted for health care practitioners who may wish to recommend mHealth technologies to their patients for pain management. Future directions for research and app development are discussed.

A Structured Review of Commercially Available Cardiac Rehabilitation mHealth Applications Using the Mobile Application Rating Scale

PURPOSE

This study systematically evaluated the quality and functionalities of patient-facing, commercially available mobile health (mHealth) apps for cardiac rehabilitation (CR).

METHODS

We performed our search in two of the most widely used commercial mobile app stores: Apple iTunes Appstore and Google Play Store (Android apps). Six search terms were used to query relevant CR apps: "cardiac rehabilitation," "heart disease and remote therapy," "heart failure exercise," "heart therapy and cardiac recovery," "cardiac recovery," and "heart therapy." App quality was evaluated using the Mobile Application Rating Scale (MARS). App functionality was evaluated using the IQVIA functionality scale, and app content was evaluated against the American Heart Association guidelines for CR. Apps meeting our inclusion criteria were downloaded and evaluated by two to three reviewers, and interclass correlations between reviewers were calculated.

RESULTS

We reviewed 3121 apps and nine apps met our inclusion criteria. On average, the apps scored a 3.0 on the MARS (5-point Likert scale) for overall quality. The two top-ranking mHealth apps for CR for all three quality, functionality, and consistency with evidence-based guidelines were My Cardiac Coach and Love My Heart for Women, both of which scored ≥4.0 for behavior change.

CONCLUSION

Overall, the quality and functionality of free apps for mobile CR was high, with two apps performing the best across all three quality categories. High-quality CR apps are available that can expand access to CR for patients with cardiovascular disease.

A Novel Method for Evaluating Mobile Apps (App Rating Inventory): Development Study

Background

Selecting and integrating health-related apps into patient care is impeded by the absence of objective guidelines for identifying high-quality apps from the many thousands now available.

Objective

This study aimed to evaluate the App Rating Inventory, which was developed by the Defense Health Agency’s Connected Health branch, to support clinical decisions regarding app selection and evaluate medical and behavioral apps.

Methods

To enhance the tool’s performance, eliminate item redundancy, reduce scoring system subjectivity, and ensure a broad application of App Rating Inventory–derived results, inventory development included 3 rounds of validation testing and 2 trial periods conducted over a 6-month interval. The development focused on content validity testing, dimensionality (ie, whether the tool’s criteria performed as operationalized), factor and commonality analysis, and interrater reliability (reliability scores improved from 0.62 to 0.95 over the course of development).

Results

The development phase culminated in a review of 248 apps for a total of 6944 data points and a final 28-item, 3-category app rating system. The App Rating Inventory produces scores for the following three categories: evidence (6 items), content (11 items), and customizability (11 items). The final (fourth) metric is the total score, which constitutes the sum of the 3 categories. All 28 items are weighted equally; no item is considered more (or less) important than any other item. As the scoring system is binary (either the app contains the feature or it does not), the ratings’ results are not dependent on a rater’s nuanced assessments.

Conclusions

Using predetermined search criteria, app ratings begin with an environmental scan of the App Store and Google Play. This first step in market research funnels hundreds of apps in a given disease category down to a manageable top 10 apps that are, thereafter, rated using the App Rating Inventory. The category and final scores derived from the rating system inform the clinician about whether an app is evidence informed and easy to use. Although a rating allows a clinician to make focused decisions about app selection in a context where thousands of apps are available, clinicians must weigh the following factors before integrating apps into a treatment plan: clinical presentation, patient engagement and preferences, available resources, and technology expertise.

Review of Mobile Apps for Women With Anxiety in Pregnancy: Maternity Care Professionals' Guide to Locating and Assessing Anxiety Apps

Background

Mental health and pregnancy apps are widely available and have the potential to improve health outcomes and enhance women’s experience of pregnancy. Women frequently access digital information throughout their pregnancy. However, health care providers and women have little information to guide them toward potentially helpful or effective apps.

Objective

This review aimed to evaluate a methodology for systematically searching and reviewing commercially available apps that support pregnant women with symptoms of anxiety in order to assist maternity care professionals in identifying resources that they could recommend for these women.

Methods

A stepwise systematic approach was used to identify, select, describe, and assess the most popular and highly user-rated apps available in the United Kingdom from January to March 2021. This included developing a script-based search strategy and search process, writing evaluation criteria, and conducting a narrative description and evaluation of the selected apps.

Results

Useful search terms were identified, which included nonclinical, aspirational, and problem-based phrases. There were 39 apps selected for inclusion in the review. No apps specifically targeted women with anxiety in pregnancy. Of the 39 apps included in the review, 33 (85%) focused solely on mind-body techniques to promote relaxation, stress reduction, and psychological well-being. Only 8 of the 39 (21%) apps included in the review reported that health care professionals had contributed to app development and only 1/39 (3%) provided empirical evidence on the effectiveness and acceptability of the app. The top 12/39 (31%) apps were evaluated by 2 independent reviewers using the developed criteria and scores. There was a small negative correlation between the reviewers’ scores and app user rating scores, with higher user rating scores associated with lower reviewer scores.

Conclusions

App developers, publishers, and maternity care professionals should seek advice from women with lived experience of anxiety symptoms in pregnancy to locate, promote, and optimize the visibility of apps for pregnant women. There is a lack of resources that provide coping strategies based on current evidence for the treatment of anxiety in pregnancy. Maternity care providers are limited in their ability to locate and recommend acceptable and trustworthy apps because of the lack of information on the evidence base, development, and testing of apps. Maternity care professionals and women need access to libraries of trusted apps that have been evaluated against relevant and established criteria.

The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective

The use of mobile technology and equipment has been found to be successful in the governance of public health. In the context of the coronavirus disease 2019 (COVID-19) pandemic, mobile health (mhealth) apps are expected to play an important role in the governance of public health. This study establishes a structural equation model based on the digital content value chain framework, identifies the main values created by mhealth apps in the prevention and control of COVID-19, and surveys 500 citizens of China. The data were analyzed using an independent t-test and partial least squares structural equations (PLS-SEM). The results showed that people who use mhealth apps are more satisfied with public health governance than those who do not; the healthcare assurance value of mhealth apps and healthcare confidence positively influence the interaction between users and mhealth app functions, the interaction with information, and the interaction with doctors to improve users’ satisfaction with public health governance; and the parasocial relationships between doctors and users of mhealth apps positively affect the interactions between users and doctors to improve users’ satisfaction with public health governance. This study confirms the potential of mhealth apps toward improving public health governance during the COVID-19 pandemic from a new perspective and provides a new theoretical basis whereby mobile technology can contribute toward improving public health governance.

mHealth Solutions for Perinatal Mental Health: Scoping Review and Appraisal Following the mHealth Index and Navigation Database Framework

Background

The ever-increasing pressure on health care systems has resulted in the underrecognition of perinatal mental disorders. Digital mental health tools such as apps could provide an option for accessible perinatal mental health screening and assessment. However, there is a lack of information regarding the availability and features of perinatal app options.

Objective

This study aims to evaluate the current state of diagnostic and screening apps for perinatal mental health available on the Google Play Store (Android) and Apple App Store (iOS) and to review their features following the mHealth Index and Navigation Database framework.

Methods

Following a scoping review approach, the Apple App Store and Google Play Store were systematically searched to identify perinatal mental health assessment apps. A total of 14 apps that met the inclusion criteria were downloaded and reviewed in a standardized manner using the mHealth Index and Navigation Database framework. The framework comprised 107 questions, allowing for a comprehensive assessment of app origin, functionality, engagement features, security, and clinical use.

Results

Most apps were developed by for-profit companies (n=10), followed by private individuals (n=2) and trusted health care companies (n=2). Out of the 14 apps, 3 were available only on Android devices, 4 were available only on iOS devices, and 7 were available on both platforms. Approximately one-third of the apps (n=5) had been updated within the last 180 days. A total of 12 apps offered the Edinburgh Postnatal Depression Scale in its original version or in rephrased versions. Engagement, input, and output features included reminder notifications, connections to therapists, and free writing features. A total of 6 apps offered psychoeducational information and references. Privacy policies were available for 11 of the 14 apps, with a median Flesch-Kincaid reading grade level of 12.3. One app claimed to be compliant with the Health Insurance Portability and Accountability Act standards and 2 apps claimed to be compliant with General Data Protection Regulation. Of the apps that could be accessed in full (n=10), all appeared to fulfill the claims stated in their description. Only 1 app referenced a relevant peer-reviewed study. All the apps provided a warning for use, highlighting that the mental health assessment result should not be interpreted as a diagnosis or as a substitute for medical care. Only 3 apps allowed users to export or email their mental health test results.

Conclusions

These results indicate that there are opportunities to improve perinatal mental health assessment apps. To this end, we recommend focusing on the development and validation of more comprehensive assessment tools, ensuring data protection and safety features are adequate for the intended app use, and improving data sharing features between users and health care professionals for timely support.

ACCU3RATE: A mobile health application rating scale based on user reviews

Background

Over the last decade, mobile health applications (mHealth App) have evolved exponentially to assess and support our health and well-being.

Objective

This paper presents an Artificial Intelligence (AI)-enabled mHealth app rating tool, called ACCU³RATE, which takes multidimensional measures such as user star rating, user review and features declared by the developer to generate the rating of an app. However, currently, there is very little conceptual understanding on how user reviews affect app rating from a multi-dimensional perspective. This study applies AI-based text mining technique to develop more comprehensive understanding of user feedback based on several important factors, determining the mHealth app ratings.

Method

Based on the literature, six variables were identified that influence the mHealth app rating scale. These factors are user star rating, user text review, user interface (UI) design, functionality, security and privacy, and clinical approval. Natural Language Toolkit package is used for interpreting text and to identify the App users’ sentiment. Additional considerations were accessibility, protection and privacy, UI design for people living with physical disability. Moreover, the details of clinical approval, if exists, were taken from the developer’s statement. Finally, we fused all the inputs using fuzzy logic to calculate the new app rating score.

Results and conclusions

ACCU³RATE concentrates on heart related Apps found in the play store and App gallery. The findings indicate the efficacy of the proposed method as opposed to the current device scale. This study has implications for both App developers and consumers who are using mHealth Apps to monitor and track their health. The performance evaluation shows that the proposed mHealth scale has shown excellent reliability as well as internal consistency of the scale, and high inter-rater reliability index. It has also been noticed that the fuzzy based rating scale, as in ACCU³RATE, matches more closely to the rating performed by experts.

A process for reviewing mental health apps: Using the One Mind PsyberGuide Credibility Rating System

Objective

Given the increasing number of publicly available mental health apps, we need independent advice to guide adoption. This paper discusses the challenges and opportunities of current mental health app rating systems and describes the refinement process of one prominent system, the One Mind PsyberGuide Credibility Rating Scale (PGCRS).

Methods

PGCRS Version 1 was developed in 2013 and deployed for 7 years, during which time a number of limitations were identified. Version 2 was created through multiple stages, including a review of evaluation guidelines and consumer research, input from scientific experts, testing, and evaluation of face validity. We then re-reviewed 161 mental health apps using the updated rating scale, investigated the reliability and discrepancy of initial scores, and updated ratings on the One Mind PsyberGuide public app guide.

Results

Reliabilities across the scale's 9 items ranged from -0.10 to 1.00, demonstrating that some characteristics of apps are more difficult to rate consistently. The average overall score of the 161 reviewed mental health apps was 2.51/5.00 (range 0.33-5.00). Ratings were not strongly correlated with app store star ratings, suggesting that credibility scores provide different information to what is contained in star ratings.

Conclusion

PGCRS summarizes and weights available information in 4 domains: intervention specificity, consumer ratings, research, and development. Final scores are created through an iterative process of initial rating and consensus review. The process of updating this rating scale and integrating it into a procedure for evaluating apps demonstrates one method for determining app quality.

Quality evaluation of stress, anxiety and depression apps for COVID-19

Background

COVID-19 has caused increased stress, anxiety and depression with increased barriers to treatment. Mobile apps offer a potential solution, but there is no information on the quality of such apps recommended for COVID-19. This study aims to evaluate the quality of stress, anxiety and depression apps recommended for COVID-19.

Methods

A search was conducted to identify relevant apps on the iOS and Android platforms. 44 apps were evaluated using the Mobile App Rating Scale (MARS), and the American Psychiatric Association's app evaluation model for data privacy and security.

Results

Overall quality scores of iOS and Android apps were 3.69 ± 0.43 and 3.66 ± 0.47. Thirty percent had good/excellent overall scores. In general, the iOS and Android versions of the apps scored best for functionality (4.21 ± 0.48, 4.12 ± 0.53), followed by aesthetics (3.84 ± 0.50, 3.78 ± 0.56), information (3.39 ± 0.54, 3.40 ± 0.60), and engagement (3.31 ± 0.81, 3.34 ± 0.84). Over half (59%) shared personal information with third parties and 14% were compliant with data protection standards.

Limitations

Only free apps available in Singapore were evaluated. Our results are time sensitive due to addition, removal, and update of apps in the app stores, thus our results should be extrapolated with caution to apps from other countries and paid apps.

Conclusion

Apps that addressed all three conditions had the highest overall quality scores. The top ranked apps (Sanvello, Woebot, Happify, Youper, Bloom) were of good quality, but majority were of acceptable quality and had room for improvement. App developers are encouraged to use our findings to improve and develop better quality apps.

Considerations of diversity, equity, and inclusion in mental health apps: A scoping review of evaluation frameworks

Mental health applications or apps are a promising approach to reduce the pervasive unmet mental health need observed among marginalized groups. However, these groups face unique challenges to engage in and benefit from these interventions, and apps that consider issues of diversity, equity, and inclusion (DEI) may be better positioned to reach them. App evaluation frameworks emerge as crucial tools for researchers, clinicians, and users to select evidence-based apps. Nevertheless, it is unclear the extent to which existing assessment tools capture DEI factors. The present scoping review identified 68 studies that employed one or more assessment tools to evaluate a mental health app, leading to 44 unique app evaluation frameworks. Results showed that most frameworks were developed after 2015, and only 58% of them considered at least one DEI criterion. Frameworks that performed exceptionally well were those that combined multiple standardized and validated measures. This lack of consideration for DEI variables may limit the ability of app-based interventions to serve marginalized communities, or even worse, create new disparities. Therefore, we provide recommendations to improve current app evaluation frameworks' cultural robustness and clinical utility, maximizing their effectiveness when working with individuals from marginalized communities.

A feature-oriented analysis of developers' descriptions and user reviews of top mHealth applications for diabetes and hypertension

BACKGROUND AND OBJECTIVE

Diabetes and hypertension are two prevalent and related chronic conditions. To inform the design and development of mobile health applications (mHealth apps) for people living with multiple chronic conditions, this paper examines features mentioned in developers' descriptions and user reviews of mHealth apps, along with users' attitudes toward associated features.

MATERIALS AND METHODS

Eleven top apps for diabetes and hypertension were identified from Google Play as of January 2020. Based on a stratified sampling strategy, 1,100 user reviews were selected to form the final dataset. Developers' descriptions were also collected for analysis. Using the grounded theory approach, we developed a feature-oriented coding scheme, which was used to identify three levels of features mentioned in app descriptions and user reviews: feature group (the highest level), feature type (the second level), and individual feature (the lowest level). Users' attitudes toward app features mentioned in user reviews were also analyzed.

RESULTS

Most top-rated apps for diabetes and hypertension under study were multifeatured, incorporating self-management, information sharing, and decision support features. At the feature-group level, most informative user reviews commented on features related to self-management, followed by decision support and information sharing. The four most frequently mentioned feature types were data entry, data export/import, data visualization, and assessment. Users expressed overwhelming positive attitudes toward app features across all feature categories. Based on users' assessments of existing features and requests for additional features, design implications for app development are provided.

CONCLUSIONS

Despite the diversity of app features provided by mHealth apps and users' primarily positive attitudes toward existing app features, more comprehensive and personalized features are expected by app users to satisfy their health needs. Beyond identifying app features in user reviews, future research may seek more in-depth feedback from real-life patients for app development and design using methods like interviews and focus groups, to further enhance the overall quality of relevant mHealth apps to better support users.

Assessing Apps for Health Care Workers Using the ISYScore-Pro Scale: Development and Validation Study

Background

The presence of mobile phone and smart devices has allowed for the use of mobile apps to support patient care. However, there is a paucity in our knowledge regarding recommendations for mobile apps specific to health care professionals.

Objective

The aim of this study is to establish a validated instrument to assess mobile apps for health care providers and health systems. Our objective is to create and validate a tool that evaluates mobile health apps aimed at health care professionals based on a trust, utility, and interest scale.

Methods

A five-step methodology framework guided our approach. The first step consisted of building a scale to evaluate apps for health care professionals based on a literature review. This was followed with expert panel validation through a Delphi method of (rated) web-based questionnaires to empirically evaluate the inclusion and weight of the indicators identified through the literature review. Repeated iterations were followed until a consensus greater than 75% was reached. The scale was then tested using a pilot to assess reliability. Interrater agreement of the pilot was measured using a weighted Cohen kappa.

Results

Using a literature review, a first draft of the scale was developed. This was followed with two Delphi rounds between the local research group and an external panel of experts. After consensus was reached, the resulting ISYScore-Pro 17-item scale was tested. A total of 280 apps were originally identified for potential testing (140 iOS apps and 140 Android apps). These were categorized using International Statistical Classification of Diseases, Tenth Revision. Once duplicates were removed and they were downloaded to confirm their specificity to the target audience (ie, health care professionals), 66 remained. Of these, only 18 met the final criteria for inclusion in validating the ISYScore-Pro scale (interrator reliabilty 92.2%; kappa 0.840, 95% CI 0.834-0.847; P<.001).

Conclusions

We have developed a reproducible methodology to objectively evaluate mobile health apps targeted to health care professionals and providers, the ISYScore-Pro scale. Future research will be needed to adapt the scale to other languages and across other domains (eg, legal compliance or security).

User Engagement and Usability of Suicide Prevention Apps: Systematic Search in App Stores and Content Analysis

Background

People with suicidal thoughts are more inclined to seek technology-delivered interventions than in-person forms of treatment, making mobile apps for suicide prevention an ideal platform for treatment delivery. This review examines apps designed for suicide prevention, with a specific focus on user engagement.

Objective

This study aims to update the literature and broadly evaluate the landscape of mobile health apps for suicide prevention; examine apps with key features and primary approaches to suicide prevention; and systematically evaluate the engagement, functionality, aesthetics, and information of the apps.

Methods

All apps related to suicidal thoughts and behaviors were identified in the Google Play and iOS app stores and were systematically reviewed for their content and quality. The mobile app rating scale (MARS) was used to evaluate app usability and engagement.

Results

Of the 66 apps identified, 42 (64%) were specifically designed for people with suicidal ideation, and 59 (89%) had at least one best practice feature for suicide risk reduction. The mean overall MARS score of all apps was 3.5 (range 2.1-4.5), with 83% (55/66) of apps having a minimum acceptability score of 3. The total MARS score was not associated with the user app rating (r=−0.001; P=.99) or the number of features (r=0.24; P=.09).

Conclusions

This study identified many usable and engaging apps in app stores designed for suicide prevention. However, there are only limited apps for clinicians. Thus, mobile apps for suicide prevention should be carefully developed and clinically evaluated.

A Review on Strategies for Data Collection, Reflection, and Communication in Eating Disorder Apps

Eating disorders (EDs) constitute a mental illness with the highest mortality. Today, mobile health apps provide promising means to ED patients for managing their condition. Apps enable users to monitor their eating habits, thoughts, and feelings, and offer analytic insights for behavior change. However, not only have scholars critiqued the clinical validity of these apps, their underlying design principles are not well understood. Through a review of 34 ED apps, we uncovered 11 different data types ED apps collect, and 9 strategies they employ to support collection and reflection. Drawing upon personal health informatics and visualization frameworks, we found that most apps did not adhere to best practices on what and how data should be collected from and reflected to users, or how data-driven insights should be communicated. Our review offers suggestions for improving the design of ED apps such that they can be useful and meaningful in ED recovery.

Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool

Background

Despite the increasing number of mobile health applications, the validity of their content is understudied. The objective of this study was to rate the content of HIV/AIDS-related mobile applications and to determine the extent to which evidence-based medicine is being incorporated into their content using a new tool called the Evidence-based content rating tool of mobile health applications (EBCRT-mHealth).

Methods

All available HIV/AIDS-related applications in Iran from Cafe Bazaar and Google Play Store were evaluated. This study was first conducted in 2018, then after almost two years in 2021 was done again. In this study, researchers developed the EBCRT-mHealth tool to rate the content of applications based on the evidence-based medicine pyramid. Its reliability was calculated (α = 0.78), and five specialists confirmed its validity. Two reviewers independently reviewed all HIV/AIDS applications directly downloaded and installed from the Google Play Store and Cafe Bazaar.

Results

Out of 980 retrieved applications, in 2018, 85, and in 2021, 78 applications were included in the study. Only in 17 (28%) out of the 60 in 2018, and 25 (51%) in 2021 Google Play store applications the source of content information was mentioned. All Cafe Bazaar mobile applications mentioned the source of information. The mean rating of all application content in 2018 was 2.38 (SD = 0.74), and in 2021 was 2.90 (SD = 1.35) out of 5. The mean rating of the content of Cafe Bazaar applications in 2018 was 2.10 (SD = 0.49), and in 2021 was 1.94 (SD = 0.29). The mean content rating of Google Play store applications in 2018 was 2.50 (SD = 0.80) and in 2021 was 3.86 (SD = 1.18).

Conclusion

After two years, the rating of the content of HIV/AIDS-related applications available in Iran that existed in Cafe Bazaar decreased from "poor" to "inappropriate". Also, the content score of the Google Play Store applications increased from "poor" to "good". It is critical to ensure the credibility of the sources used in developing their content and removing applications with inappropriate and unreliable content from the App Stores. Also, mobile health application developers should use the highest quality information in their applications.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01498-7.

Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides

Background

There is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed.

Objective

The main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders.

Methods

In order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability.

Results

Only a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29.

Conclusions

This study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers

Objective

The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.

Materials and Methods

We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of “health apps,” “evaluation,” and “frameworks.” The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included.

Results

Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or “threshold” filter.

Discussion

There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps.

Conclusions

A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.

Assessing the quality of mobile applications in chronic disease management: a scoping review

While there has been a rapid growth of digital health apps to support chronic diseases, clear standards on how to best evaluate the quality of these evolving tools are absent. This scoping review aims to synthesize the emerging field of mobile health app quality assessment by reviewing criteria used by previous studies to assess the quality of mobile apps for chronic disease management. A literature review was conducted in September 2017 for published studies that use a set of quality criteria to directly evaluate two or more patient-facing apps supporting promote chronic disease management. This resulted in 8182 citations which were reviewed by research team members, resulting in 65 articles for inclusion. An inductive coding schema to synthesize the quality criteria utilized by included articles was developed, with 40 unique quality criteria identified. Of the 43 (66%) articles that reported resources used to support criteria selection, 19 (29%) used clinical guidelines, and 10 (15%) used behavior change theory. The most commonly used criteria included the presence of user engagement or behavior change functions (97%, n = 63) and technical features of the app such as customizability (20%, n = 13, while Usability was assessed by 24 studies (36.9%). This study highlights the significant variation in quality criteria employed for the assessment of mobile health apps. Future methods for app evaluation will benefit from approaches that leverage the best evidence regarding the clinical impact and behavior change mechanisms while more directly reflecting patient needs when evaluating the quality of apps.

Technology Evaluation and Assessment Criteria for Health Apps (TEACH-Apps): Pilot Study

BACKGROUND

Despite the emergence of app evaluation tools, there remains no well-defined process receptive to diverse local needs, rigorous standards, and current content. The need for such a process to assist in the implementation of app evaluation across all medical fields is evident. Such a process has the potential to increase stakeholder engagement and catalyze interest and engagement with present-day app evaluation models.

OBJECTIVE

This study aimed to develop and pilot test the Technology Evaluation and Assessment Criteria for Health apps (TEACH-apps).

METHODS

Tailoring a well-known implementation framework, Replicating Effective Programs, we present a new process to approach the challenges faced in implementing app evaluation tools today. As a culmination of our experience implementing this process and feedback from stakeholders, we present the four-part process to aid the implementation of mobile health technology. This paper outlines the theory, evidence, and initial versions of the process.

RESULTS

The TEACH-apps process is designed to be broadly usable and widely applicable across all fields of health. The process comprises four parts: (1) preconditions (eg, gathering apps and considering local needs), (2) preimplementation (eg, customizing criteria and offering digital skills training), (3) implementation (eg, evaluating apps and creating educational handouts), and (4) maintenance and evolution (eg, repeating the process every 90 days and updating content). TEACH-apps has been tested internally at our hospital, and there is growing interest in partnering health care facilities to test the system at their sites.

CONCLUSIONS

This implementation framework introduces a process that equips stakeholders, clinicians, and users with the foundational tools to make informed decisions around app use and increase app evaluation engagement. The application of this process may lead to the selection of more culturally appropriate and clinically relevant tools in health care.

Theme Trends and Knowledge Structure on Mobile Health Apps: Bibliometric Analysis

Background

Due to the widespread and unprecedented popularity of mobile phones, the use of digital medicine and mobile health apps has seen significant growth. Mobile health apps have tremendous potential for monitoring and treating diseases, improving patient care, and promoting health.

Objective

This paper aims to explore research trends, coauthorship networks, and the research hot spots of mobile health app research.

Methods

Publications related to mobile health apps were retrieved and extracted from the Web of Science database with no language restrictions. Bibliographic Item Co-Occurrence Matrix Builder was employed to extract bibliographic information (publication year and journal source) and perform a descriptive analysis. We then used the VOSviewer (Leiden University) tool to construct and visualize the co-occurrence networks of researchers, research institutions, countries/regions, citations, and keywords.

Results

We retrieved 2802 research papers on mobile health apps published from 2000 to 2019. The number of annual publications increased over the past 19 years. JMIR mHealth and uHealth (323/2802, 11.53%), Journal of Medical Internet Research (106/2802, 3.78%), and JMIR Research Protocols (82/2802, 2.93%) were the most common journals for these publications. The United States (1186/2802, 42.33%), England (235/2802, 8.39%), Australia (215/2802, 7.67%), and Canada (112/2802, 4.00%) were the most productive countries of origin. The University of California San Francisco, the University of Washington, and the University of Toronto were the most productive institutions. As for the authors’ contributions, Schnall R, Kuhn E, Lopez-Coronado M, and Kim J were the most active researchers. The co-occurrence cluster analysis of the top 100 keywords forms 5 clusters: (1) the technology and system development of mobile health apps; (2) mobile health apps for mental health; (3) mobile health apps in telemedicine, chronic disease, and medication adherence management; (4) mobile health apps in health behavior and health promotion; and (5) mobile health apps in disease prevention via the internet.

Conclusions

We summarize the recent advances in mobile health app research and shed light on their research frontier, trends, and hot topics through bibliometric analysis and network visualization. These findings may provide valuable guidance on future research directions and perspectives in this rapidly developing field.

Adherence to Established Treatment Guidelines Among Unguided Digital Interventions for Depression: Quality Evaluation of 28 Web-Based Programs and Mobile Apps

Background

Web-based interventions for depression have been widely tested for usability and functioning. However, the few studies that have addressed the therapeutic quality of these interventions have mainly focused on general aspects without consideration of specific quality factors related to particular treatment components. Clinicians and scientists are calling for standardized assessment criteria for web-based interventions to enable effective and trustworthy patient care. Therefore, an extensive evaluation of web-based interventions at the level of individual treatment components based on therapeutic guidelines and manuals is needed.

Objective

The objective of this study was to evaluate the quality of unguided web-based interventions for depression at the level of individual treatment components based on their adherence to current gold-standard treatment guidelines and manuals.

Methods

A comprehensive online search of popular app stores and search engines in January 2018 revealed 11 desktop programs and 17 smartphone apps that met the inclusion criteria. Programs and apps were included if they were available for German users, interactive, unguided, and targeted toward depression. All programs and apps were tested by three independent researchers following a standardized procedure with a predefined symptom trajectory. During the testing, all web-based interventions were rated with a standardized list of criteria based on treatment guidelines and manuals for depression.

Results

Overall interrater reliability for all raters was substantial with an intraclass correlation coefficient of 0.73 and Gwet AC1 value of 0.80. The main features of web-based interventions included mood tracking (24/28, 86%), psychoeducation (21/28, 75%), cognitive restructuring (21/28, 75%), crisis management (20/28, 71%), behavioral activation (19/29, 68%), and relaxation training (18/28, 64%). Overall, therapeutic meaningfulness was rated higher for desktop programs (mean 4.13, SD 1.17) than for smartphone apps (mean 2.92, SD 1.46).

Conclusions

Although many exercises from manuals are included in web-based interventions, the necessary therapeutic depth of the interventions is often not reached, and risk management is frequently lacking. There is a need for further research targeting general principles for the development and evaluation of therapeutically sound web-based interventions for depression.

Longitudinal trends in the quality, effectiveness and attributes of highly rated smartphone health apps

Background

While there are numerous mental health apps on the market today, less is known about their safety and quality. This study aims to offer a longitudinal perspective on the nature of high visibility apps for common mental health and physical health conditions.

Methods

In July 2019, we selected the 10 top search-returned apps in the Apple App Store and Android Google Play Store using six keyword terms: depression, anxiety, schizophrenia, addiction, high blood pressure and diabetes. Each app was downloaded by two authors and reviewed by a clinician, and the app was coded for features, functionality, claims, app store properties, and other properties.

Results

Compared with 1 year prior, there were few statistically significant changes in app privacy policies, evidence and features. However, there was a high rate of turnover with only 34 (57%) of the apps from the Apple’s App Store and 28 (47%) from the Google Play Store remaining in the 2019 top 10 search compared with the 2018 search.

Discussion

Although there was a high turnover of top search-returned apps between 2018 and 2019, we found that there were few significant changes in features, privacy, medical claims and other properties. This suggests that, although the highly visible and available apps are changing, there were no significant improvements in app quality or safety.

Existing Mobile Phone Apps for Self-Care Management of People With Alzheimer Disease and Related Dementias: Systematic Analysis

BACKGROUND

Alzheimer disease and related dementias (AD/RD) are progressive neurocognitive disorders that currently affect approximately 50 million people worldwide. Mobile phone apps have been well-integrated into daily lives and can be used to deliver and promote health care. There is an increase in the use of technology to provide care and support to AD/RD patients and their families.

OBJECTIVE

This study aimed to review apps designed for AD/RD patients and analyze the benefits of, and challenges to, such technological solutions.

METHODS

A systematic approach was applied to review the availability, content, features, and quality of mobile phone apps to support self-care among AD/RD patients.

RESULTS

The initial search for this review was conducted in January 2019, and the screening and analysis of the included apps were completed in May 2019. A total of 14 apps were included from an initial search of 245 apps. The top 3 features were alert (9/14, 64%), self-care tips (6/14, 42%), and social networking capacity (5/14, 35%). On average, the readability of the apps was a tenth-grade reading level (SD 3.06). The overall quality was 3.71 out of 5 (SD 1.37).

CONCLUSIONS

Our findings suggest that currently available apps for AD/RD patients may not meet complex needs and may be challenging to use, given the possible impaired communication ability associated with AD/RD. Therefore, high-quality apps need to be developed and rigorously evaluated for feasibility and efficacy.

Functionality of Top-Rated Mobile Apps for Depression: Systematic Search and Evaluation

BACKGROUND

In the last decade, there has been a proliferation of mobile apps claiming to support the needs of people living with depression. However, it is unclear what functionality is actually provided by apps for depression, or for whom they are intended.

OBJECTIVE

This paper aimed to explore the key features of top-rated apps for depression, including descriptive characteristics, functionality, and ethical concerns, to better inform the design of apps for depression.

METHODS

We reviewed top-rated iPhone OS (iOS) and Android mobile apps for depression retrieved from app marketplaces in spring 2019. We applied a systematic analysis to review the selected apps, for which data were gathered from the 2 marketplaces and through direct use of the apps. We report an in-depth analysis of app functionality, namely, screening, tracking, and provision of interventions. Of the initially identified 482 apps, 29 apps met the criteria for inclusion in this review. Apps were included if they remained accessible at the moment of evaluation, were offered in mental health-relevant categories, received a review score greater than 4.0 out of 5.0 by more than 100 reviewers, and had depression as a primary target.

RESULTS

The analysis revealed that a majority of apps specify the evidence base for their intervention (18/29, 62%), whereas a smaller proportion describes receiving clinical input into their design (12/29, 41%). All the selected apps are rated as suitable for children and adolescents on the marketplace, but 83% (24/29) do not provide a privacy policy consistent with their rating. The findings also show that most apps provide multiple functions. The most commonly implemented functions include provision of interventions (24/29, 83%) either as a digitalized therapeutic intervention or as support for mood expression; tracking (19/29, 66%) of moods, thoughts, or behaviors for supporting the intervention; and screening (9/29, 31%) to inform the decision to use the app and its intervention. Some apps include overtly negative content.

CONCLUSIONS

Currently available top-ranked apps for depression on the major marketplaces provide diverse functionality to benefit users across a range of age groups; however, guidelines and frameworks are still needed to ensure users' privacy and safety while using them. Suggestions include clearly defining the age of the target population and explicit disclosure of the sharing of users' sensitive data with third parties. In addition, we found an opportunity for apps to better leverage digital affordances for mitigating harm, for personalizing interventions, and for tracking multimodal content. The study further demonstrated the need to consider potential risks while using depression apps, including the use of nonvalidated screening tools, tracking negative moods or thinking patterns, and exposing users to negative emotional expression content.

Regulating digital health technologies with transparency: the case for dynamic and multi-stakeholder evaluation

BACKGROUND

The prevalence of smartphones today, paired with the increasing precision and therapeutic potential of digital capabilities, offers unprecedented opportunity in the field of digital medicine. Smartphones offer novel accessibility, unique insights into physical and cognitive behavior, and diverse resources designed to aid health. Many of these digital resources, however, are developed and shared at a faster rate than they can be assessed for efficacy, safety, and security-presenting patients and clinicians with the challenge of distinguishing helpful tools from harmful ones.

MAIN TEXT

Leading regulators, such as the FDA in the USA and the NHS in the UK, are working to evaluate the influx of mobile health applications entering the market. Efforts to regulate, however, are challenged by the need for more transparency. They require real-world data on the actual use, effects, benefits, and harms of these digital health tools. Given rapid product cycles and frequent updates, even the most thorough evaluation is only as accurate as the data it is based on.

CONCLUSIONS

In this debate piece, we propose a complementary approach to ongoing efforts via a dynamic self-certification checklist. We outline how simple self-certification, validated or challenged by app users, would enhance transparency, engage diverse stakeholders in meaningful education and learning, and incentivize the design of safe and secure medical apps.

Criteria for assessing the quality of mHealth apps: a systematic review

Objective

Review the existing studies including an assessment tool/method to assess the quality of mHealth apps; extract their criteria; and provide a classification of the collected criteria.

Methods

In accordance with the PRISMA statement, a literature search was conducted in MEDLINE, EMBase, ISI and Scopus for English language citations published from January 1, 2008 to December 22, 2016 for studies including tools or methods for quality assessment of mHealth apps. Two researchers screened the titles and abstracts of all retrieved citations against the inclusion and exclusion criteria. The full text of relevant papers was then individually examined by the same researchers. A senior researcher resolved eventual disagreements and confirmed the relevance of all included papers. The authors, date of publication, subject fields of target mHealth apps, development method, and assessment criteria were extracted from each paper. The extracted assessment criteria were then reviewed, compared, and classified by an expert panel of two medical informatics specialists and two health information management specialists.

Results

Twenty-three papers were included in the review. Thirty-eight main classes of assessment criteria were identified. These were reorganized by expert panel into 7 main classes (Design, Information/Content, Usability, Functionality, Ethical Issues, Security and Privacy, and User-perceived value) with 37 sub-classes of criteria.

Conclusions

There is a wide heterogeneity in assessment criteria for mHealth apps. It is necessary to define the exact meanings and degree of distinctness of each criterion. This will help to improve the existing tools and may lead to achieve a better comprehensive mHealth app assessment tool.

A design and evaluation framework for digital health interventions

Digital health interventions (DHIs) have the potential to help the growing number of chronic disease patients better manage their everyday lives. However, guidelines for the systematic development of DHIs are still scarce. The current work has, therefore, the objective to propose a framework for the design and evaluation of DHIs (DEDHI). The DEDHI framework is meant to support both researchers and practitioners alike from early conceptual DHI models to large-scale implementations of DHIs in the healthcare market.

Validation of an mHealth App for Depression Screening and Monitoring (Psychologist in a Pocket): Correlational Study and Concurrence Analysis

Background

Mobile health (mHealth) is a fast-growing professional sector. As of 2016, there were more than 259,000 mHealth apps available internationally. Although mHealth apps are growing in acceptance, relatively little attention and limited efforts have been invested to establish their scientific integrity through statistical validation. This paper presents the external validation of Psychologist in a Pocket (PiaP), an Android-based mental mHealth app which supports traditional approaches in depression screening and monitoring through the analysis of electronic text inputs in communication apps.

Objective

The main objectives of the study were (1) to externally validate the construct of the depression lexicon of PiaP with standardized psychological paper-and-pencil tools and (2) to determine the comparability of PiaP, a new depression measure, with a psychological gold standard in identifying depression.

Methods

College participants downloaded PiaP for a 2-week administration. Afterward, they were asked to complete 4 psychological depression instruments. Furthermore, 1-week and 2-week PiaP total scores (PTS) were correlated with (1) Beck Depression Index (BDI)-II and Center for Epidemiological Studies–Depression (CES-D) Scale for congruent construct validation, (2) Affect Balance Scale (ABS)–Negative Affect for convergent construct validation, and (3) Satisfaction With Life Scale (SWLS) and ABS–Positive Affect for divergent construct validation. In addition, concordance analysis between PiaP and BDI-II was performed.

Results

On the basis of the Pearson product-moment correlation, significant positive correlations exist between (1) 1-week PTS and CES-D Scale, (2) 2-week PTS and BDI-II, and (3) PiaP 2-week PTS and SWLS. Concordance analysis (Bland-Altman plot and analysis) suggested that PiaP’s approach to depression screening is comparable with the gold standard (BDI-II).

Conclusions

The evaluation of mental health has historically relied on subjective measurements. With the integration of novel approaches using mobile technology (and, by extension, mHealth apps) in mental health care, the validation process becomes more compelling to ensure their accuracy and credibility. This study suggests that PiaP’s approach to depression screening by analyzing electronic data is comparable with traditional and well-established depression instruments and can be used to augment the process of measuring depression symptoms.

Mobile Apps for the Care Management of Chronic Kidney and End-Stage Renal Diseases: Systematic Search in App Stores and Evaluation

Background

Numerous free and low-cost mobile apps for the care management of kidney disease have become available in recent years. Although these appear to be promising tools, they have not been evaluated comparatively based on standard mobile app metrics, and thus, limited evidence is available regarding their efficacy. This study systematically cataloged and assessed mobile apps designed to assist medication compliance and nutrition tracking that are useful to the chronic kidney disease (CKD) and the end-stage renal disease (ESRD) patients who are on dialysis.

Objective

The objective of this study was to comprehensively evaluate mobile apps used for medication compliance and nutrition tracking for possible use by CKD and ESRD patients.

Methods

A systematic review framework was applied to the search, screening, and assessment of apps identified and downloaded from the iOS and Android app stores. We selected apps using 13 relevant search terms, narrowed down based on a set of inclusion and exclusion criteria, and then used the Mobile App Rating Scale (MARS), a widely adopted app evaluation tool to assess the effectiveness of apps. The internal consistency and interrater reliability were tested using Cronbach alpha and interclass correlation coefficients (ICCs), respectively.

Results

The MARS total score had excellent internal consistency (Cronbach alpha=.90) and a moderate level of interrater reliability (2-way mixed ICC 0.65). Overall, 11 out of the 12 reviewed apps met the minimum acceptable score of 3.0 in MARS rating. The 3 apps with the highest combined scores were My Kidneys, My Health Handbook (MARS=4.68); My Food Coach (MARS=4.48); and National Kidney Foundation Malaysia (MARS=4.20). The study identified 2 general weaknesses in the existing apps: the apps fell short of accommodating advanced interactive features such as providing motivational feedback and promoting family member and caregiver participations in the app utilization.

Conclusions

The MARS rating system performed well in the app evaluation. The 3 highest ranked apps scored consistently high across the 5 dimensions specified in MARS. These apps were developed in collaboration with reputable organizations and field experts, demonstrating the importance of expert guidance in developing medical apps.

German Mobile Apps in Rheumatology: Review and Analysis Using the Mobile Application Rating Scale (MARS)

Background

Chronic rheumatic diseases need long-term treatment and professional supervision. Mobile apps promise to improve the lives of patients and physicians. In routine practice, however, rheumatology apps are largely unknown and little is known about their quality and safety.

Objective

The aim of this study was to provide an overview of mobile rheumatology apps currently available in German app stores, evaluate app quality using the Mobile Application Rating Scale (MARS), and compile brief, ready-to-use descriptions for patients and rheumatologists.

Methods

The German App Store and Google Play store were systematically searched to identify German rheumatology mobile apps for patient and physician use. MARS was used to independently assess app quality by 8 physicians, 4 using Android and 4 using iOS smartphones. Apps were randomly assigned so that 4 apps were rated by all raters and the remaining apps were rated by two Android and two iOS users. Furthermore, brief app descriptions including app developers, app categories, and features were compiled to inform potential users and developers.

Results

In total, 128 and 63 apps were identified in the German Google Play and App Store, respectively. After removing duplicates and only including apps that were available in both stores, 28 apps remained. Sixteen apps met the inclusion criteria, which were (1) German language, (2) availability in both app stores, (3) targeting patients or physicians as users, and (4) clearly including rheumatology or rheumatic diseases as subject matter. Exclusion criteria were (1) congress apps and (2) company apps with advertisements. Nine apps addressed patients and 7 apps addressed physicians. No clinical studies to support the effectiveness and safety of apps could be found. Pharmaceutical companies were the main developers of two apps. Rheuma Auszeit was the only app mainly developed by a patient organization. This app had the highest overall MARS score (4.19/5). Three out of 9 patient apps featured validated questionnaires. The median overall MARS score was 3.85/5, ranging from 2.81/5 to 4.19/5. One patient-targeted and one physician-targeted app had MARS scores >4/5. No significant rater gender or platform (iOS/Android) differences could be observed. The overall correlation between app store ratings and MARS scores was low and inconsistent between platforms.

Conclusions

To our knowledge, this is the first study that systematically identified and evaluated mobile apps in rheumatology for patients and physicians available in German app stores. We found a lack of supporting clinical studies, use of validated questionnaires, and involvement of academic developers. Overall app quality was heterogeneous. To create high-quality apps, closer cooperation led by patients and physicians is vital.

Using Health and Well-Being Apps for Behavior Change: A Systematic Search and Rating of Apps

Background

Smartphones have allowed for the development and use of apps. There is now a proliferation of mobile health interventions for physical activity, healthy eating, smoking and alcohol cessation or reduction, and improved mental well-being. However, the strength or potential of these apps to lead to behavior change remains uncertain.

Objective

The aim of this study was to review a large sample of healthy lifestyle apps at a single point in time (June to July 2018) to determine their potential for promoting health-related behavior change with a view to sharing this information with the public. In addition, the study sought to test a wide range of apps using a new scale, the App Behavior Change Scale (ABACUS).

Methods

Apps focusing on 5 major modifiable lifestyle behaviors were identified using a priori key search terms across the Australian Apple iTunes and Google Play stores. Lifestyle behavior categories were selected for their impact on health and included smoking, alcohol use, physical activity, nutrition, and mental well-being. Apps were included if they had an average user rating between 3 and 5, if they were updated in the last 18 months, if the description of the app included 2 of 4 behavior change features, and if they were in English. The selected behavior change apps were rated in 2 ways using previously developed rating scales: the Mobile App Rating Scale (MARS) for functionality and the ABACUS for potential to encourage behavior change.

Results

The initial search identified 212,352 apps. After applying the filtering criteria, 5018 apps remained. Of these, 344 were classified as behavior change apps and were reviewed and rated. Apps were given an average MARS score of 2.93 out of 5 (SD 0.58, range 1.42-4.16), indicating low-to-moderate functionality. Scores for the ABACUS ranged from 1 to 17, out of 21, with an average score of 7.8 (SD 2.8), indicating a low-to-moderate number of behavior change techniques included in apps. The ability of an app to encourage practice or rehearsal, in addition to daily activities, was the most commonly identified feature across all apps (310/344, 90.1%), whereas the second most common feature was the ability of the user to easily self-monitor behavior (289/344, 84.0%).

Conclusions

The wide variety of apps included in this 2018 study and the limited number of behavior change techniques found in many apps suggest an opportunity for improvement in app design that will promote sustained and significant lifestyle behavior change and, therefore, better health. The use of the 2 scales for the review and rating of the apps was successful and provided a method that could be replicated and tested in other behavior change areas.

Mental Health Apps in Psychiatric Treatment: A Patient Perspective on Real World Technology Usage

For many people who use mobile apps, the primary motivations are entertainment, news, gaming, social connections, or productivity. For those experiencing health problems, particularly those with chronic conditions such as psychiatric disorders, the stakes are much higher. The digital tools that they select may be the difference between improvement and decompensation or even life and death. Although there has been a wide expansion of mental health apps with promise as well as hype, the current means of researching, evaluating, and deploying effective tools have been problematic. As a means of gaining a perspective that moves beyond usability testing, surveys, and app ratings, the primary objective of this patient perspective is to question the killer app and condition-specific mentality of current mental health app development. We do this by reviewing the current mobile mental health app literature, identifying ways in which psychiatric patients use apps in their lives, and then exploring how these issues are experienced by a software engineer who has struggled with her bipolar disorder for many years. Her lived experience combined with a technology perspective offers potential avenues for using technology productively in psychiatric treatment. We believe that this responds to JMIR Publications' call for patient perspective papers and provides encouragement for patients to share their views on mental health and technology.

Evaluation of Mobile Apps Targeted to Parents of Infants in the Neonatal Intensive Care Unit: Systematic App Review

Background

Parents of preterm infants increasingly use their mobile phone to search for health information. In a recent review, websites targeted toward parents with infants in the neonatal intensive care unit (NICU) were found to have poor to moderate quality educational material; however, there is a dearth of literature regarding mobile apps for NICU parents.

Objective

This study aimed to identify and evaluate apps targeting parents of infants in the NICU for quality of information, usability, and credibility.

Methods

We systematically searched the Apple App Store and Google Play using 49 key terms (eg, “preterm infant”) from July 26 to August 18, 2017. English apps targeting NICU parents that cost less than $20 were included. Apps for health care professionals, e-books/magazines, or nonrelevant results were excluded. In total, 3 tools were used for evaluation: Mobile Application Rating Scale (MARS) to measure quality; Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-AV) to measure the app’s content usability; and Trust it or Trash It to measure credibility.

Results

The initial search yielded 6579 apps, with 49 apps eligible after title and description screening. In total, 27 apps met the eligibility criteria with 9 apps available in both app stores; of those, the app with the most recent update date was chosen to be included in the analysis. Thus, 18 unique apps were included for final analysis. Using MARS, 7 apps (7/18, 39%) received a good score on overall quality (ie, 4.0 out of 5.0), with none receiving an excellent score. In addition, 8 apps (8/18, 44%) received a PEMAT-AV score between 51% and 75% on the understandability subscale, and 8 apps (8/18, 44%) scored between 76% and 100% on the actionability subscale. Trust It or Trash It deemed 13 apps (13/18, 72%) as trash for reasons including no identification of sources or lack of current information, with only 5 (5/18, 28%) deemed trustworthy. Reviewer’s expert evaluation found 16 apps contained content that matched information provided by multiple sources; however, most apps did not meet other objective measurement items to support credibility. When comparing the MARS overall quality and subjective quality scores with trustworthiness of apps, there was no statistically significant difference. A statistically significant difference was found between the 2 MARS quality scores, indicating that, on average, apps were ranked significantly lower on subjective quality compared with overall quality measures.

Conclusions

This evaluation revealed that of the available apps targeting NICU parents, less than half should be considered as acceptable educational material. Over two-thirds of the apps were found to have issues regarding credibility and just over a quarter were considered good quality. The apps currently available for NICU parents are lacking and of concern in terms of quality and credibility.

Assessment of the Data Sharing and Privacy Practices of Smartphone Apps for Depression and Smoking Cessation

IMPORTANCE

Inadequate privacy disclosures have repeatedly been identified by cross-sectional surveys of health applications (apps), including apps for mental health and behavior change. However, few studies have assessed directly the correspondence between privacy disclosures and how apps handle personal data. Understanding the scope of this discrepancy is particularly important in mental health, given enhanced privacy concerns relating to stigma and negative impacts of inadvertent disclosure. Because most health apps fall outside government regulation, up-to-date technical scrutiny is essential for informed decision making by consumers and health care professionals wishing to prescribe health apps.

OBJECTIVE

To provide a contemporary assessment of the privacy practices of popular apps for depression and smoking cessation by critically evaluating privacy policy content and, specifically, comparing disclosures regarding third-party data transmission to actual behavior.

DESIGN AND SETTING

Cross-sectional assessment of 36 top-ranked (by app store search result ordering in January 2018) apps for depression and smoking cessation for Android and iOS in the United States and Australia. Privacy policy content was evaluated with prespecified criteria. Technical assessment of encrypted and unencrypted data transmission was performed. Analysis took place between April and June 2018.

MAIN OUTCOMES AND MEASURES

Correspondence between policies and transmission behavior observed by intercepting sent data.

RESULTS

Twenty-five of 36 apps (69%) incorporated a privacy policy. Twenty-two of 25 apps with a policy (88%) provided information about primary uses of collected data, while only 16 (64%) described secondary uses. While 23 of 25 apps with a privacy policy (92%) stated in a policy that data would be transmitted to a third party, transmission was detected in 33 of all 36 apps (92%). Twenty-nine of 36 apps (81%) transmitted data for advertising and marketing purposes or analytics to just 2 commercial entities, Google and Facebook, but only 12 of 28 (43%) transmitting data to Google and 6 of 12 (50%) transmitting data to Facebook disclosed this.

CONCLUSIONS AND RELEVANCE

Data sharing with third parties that includes linkable identifiers is prevalent and focused on services provided by Google and Facebook. Despite this, most apps offer users no way to anticipate that data will be shared in this way. As a result, users are denied an informed choice about whether such sharing is acceptable to them. Privacy assessments that rely solely on disclosures made in policies, or are not regularly updated, are unlikely to uncover these evolving issues. This may limit their ability to offer effective guidance to consumers and health care professionals.

The App Behavior Change Scale: Creation of a Scale to Assess the Potential of Apps to Promote Behavior Change

BACKGROUND

Using mobile phone apps to promote behavior change is becoming increasingly common. However, there is no clear way to rate apps against their behavior change potential.

OBJECTIVE

This study aimed to develop a reliable, theory-based scale that can be used to assess the behavior change potential of smartphone apps.

METHODS

A systematic review of all studies purporting to investigate app's behavior change potential was conducted. All scales and measures from the identified studies were collected to create an item pool. From this item pool, 3 health promotion exerts created the App Behavior Change Scale (ABACUS). To test the scale, 70 physical activity apps were rated to provide information on reliability.

RESULTS

The systematic review returned 593 papers, the abstracts and titles of all were reviewed, with the full text of 77 papers reviewed; 50 papers met the inclusion criteria. From these 50 papers, 1333 questions were identified. Removing duplicates and unnecessary questions left 130 individual questions, which were then refined into the 21-item scale. The ABACUS demonstrates high percentage agreement among reviewers (over 80%), with 3 questions scoring a Krippendorff alpha that would indicate agreement and a further 7 came close with alphas >.5. The scale overall reported high interrater reliability (2-way mixed interclass coefficient=.92, 95% CI 0.81-0.97) and high internal consistency (Cronbach alpha=.93).

CONCLUSIONS

The ABACUS is a reliable tool that can be used to determine the behavior change potential of apps. This instrument fills a gap by allowing the evaluation of a large number of apps to be standardized across a range of health categories.

Understanding the quality, effectiveness and attributes of top-rated smartphone health apps

Objective

This study aimed to understand the attributes of popular apps for mental health and comorbid medical conditions, and how these qualities relate to consumer ratings, app quality and classification by the WHO health app classification framework.

Methods

We selected the 10 apps from the Apple iTunes store and the US Android Google Play store on 20 July 2018 from six disease states: depression, anxiety, schizophrenia, addiction, diabetes and hypertension. Each app was downloaded by two authors who provided information on the apps’ attributes, functionality, interventions, popularity, scientific backing and WHO app classification rating.

Results

A total of 120 apps were examined. Although none of these apps had Food and Drug Administration marketing approval, nearly 50% made claims that appeared medical. Most apps offered a similar type of services with 87.5% assigned WHO classification 1.4.2 ‘self-monitoring of health or diagnostic data by a client’ or 1.6.1 ‘client look-up of health information’. The ‘last updated’ attribute was highly correlated with a quality rating of the app although no apps features (eg, uses Global Positioning System, reminders and so on) were.

Conclusion

Due to the heterogeneity of the apps, we were unable to define a core set of features that would accurately assess app quality. The number of apps making unsupported claims combined with the number of apps offering questionable content warrants a cautious approach by both patients and clinicians in selecting safe and effective ones.

Clinical Implications

‘Days since last updated’ offers a useful and easy clinical screening test for health apps, regardless of the condition being examined.

Smartphone-Based Physical Activity Telecoaching in Chronic Obstructive Pulmonary Disease: Mixed-Methods Study on Patient Experiences and Lessons for Implementation

Background

Telecoaching approaches can enhance physical activity (PA) in patients with chronic obstructive pulmonary disease (COPD). However, their effectiveness is likely to be influenced by intervention-specific characteristics.

Objective

This study aimed to assess the acceptability, actual usage, and feasibility of a complex PA telecoaching intervention from both patient and coach perspectives and link these to the effectiveness of the intervention.

Methods

We conducted a mixed-methods study based on the completers of the intervention group (N=159) included in an (effective) 12-week PA telecoaching intervention. This semiautomated telecoaching intervention consisted of a step counter and a smartphone app. Data from a project-tailored questionnaire (quantitative data) were combined with data from patient interviews and a coach focus group (qualitative data) to investigate patient and coach acceptability, actual usage, and feasibility of the intervention. The degree of actual usage of the smartphone and step counter was also derived from app data. Both actual usage and perception of feasibility were linked to objectively measured change in PA.

Results

The intervention was well accepted and perceived as feasible by all coaches present in the focus group as well by patients, with 89.3% (142/159) of patients indicating that they enjoyed taking part. Only a minority of patients (8.2%; 13/159) reported that they found it difficult to use the smartphone. Actual usage of the step counter was excellent, with patients wearing it for a median (25th-75th percentiles) of 6.3 (5.8-6.8) days per week, which did not change over time (P=.98). The smartphone interface was used less frequently and actual usage of all daily tasks decreased significantly over time (P<.001). Patients needing more contact time had a smaller increase in PA, with mean (SD) of +193 (SD 2375) steps per day, +907 (SD 2306) steps per day, and +1489 (SD 2310) steps per day in high, medium, and low contact time groups, respectively; P for-trend=.01. The overall actual usage of the different components of the intervention was not associated with change in step count in the total group (P=.63).

Conclusions

The 12-week semiautomated PA telecoaching intervention was well accepted and feasible for patients with COPD and their coaches. The actual usage of the step counter was excellent, whereas actual usage of the smartphone tasks was lower and decreased over time. Patients who required more contact experienced less PA benefits.

Trial Registration

ClinicalTrials.gov NCT02158065; http://clinicaltrials.gov/ct2/show/NCT02158065 (Archived by WebCite at http://www.webcitation.org/73bsaudy9)

Examining Predictors of Real-World User Engagement with Self-Guided eHealth Interventions: Analysis of Mobile Apps and Websites Using a Novel Dataset

Background

The literature suggests that the product design of self-guided electronic health (eHealth) interventions impacts user engagement. Traditional trial settings, however, do not enable the examination of these relationships in real-world use.

Objective

This study aimed to examine whether the qualities of product design, research evidence, and publicly available data predict real-world user engagement with mobile and Web-based self-guided eHealth interventions.

Methods

This analysis included self-guided mobile and Web-based eHealth interventions available to the public—with their qualities assessed using the Enlight suite of scales. Scales included Usability, Visual Design, User Engagement, Content, Therapeutic Persuasiveness, Therapeutic Alliance, Credibility, and Research Evidence. Behavioral data on real-world usage were obtained from a panel that provides aggregated nonpersonal information on user engagement with websites and mobile apps, based on a time window of 18 months that was set between November 1, 2016 and April 30, 2018. Real-world user engagement variables included average usage time (for both mobile apps and websites) and mobile app user retention 30 days after download.

Results

The analysis included 52 mobile apps (downloads median 38,600; interquartile range [IQR] 116,000) and 32 websites (monthly unique visitors median 5689; IQR 30,038). Results point to moderate correlations between Therapeutic Persuasiveness, Therapeutic Alliance, and the 3 user engagement variables (.31≤rs≤.51; Ps≤.03). Visual Design, User Engagement, and Content demonstrated similar degrees of correlation with mobile app engagement variables (.25≤rs≤.49; Ps≤.04) but not with average usage time of Web-based interventions. Positive correlations were also found between the number of reviews on Google Play and average app usage time (r=.58; P<.001) and user retention after 30 days (r=.23; P=.049). Although several product quality ratings were positively correlated with research evidence, the latter was not significantly correlated with real-world user engagement. Hierarchical stepwise regression analysis revealed that either Therapeutic Persuasiveness or Therapeutic Alliance explained 15% to 26% of user engagement variance. Data on Google Play (number of reviews) explained 15% of the variance of mobile app usage time above Enlight ratings; however, publicly available data did not significantly contribute to explaining the variance of the other 2 user-engagement variables.

Conclusions

Results indicate that the qualities of product design predict real-world user engagement with eHealth interventions. The use of real-world behavioral datasets is a novel way to learn about user behaviors, creating new avenues for eHealth intervention research.

QUALITY AND USE OF FREE SMOKING CESSATION APPS FOR SMARTPHONES

OBJECTIVES

Smartphone Apps are one of the tools available to support patients who wish to quit smoking. Content analysis studies have indicated multiple deficiencies within these Apps including minimal use of evidence-based research and Nicotine Dependence Treatment Provider (NDTP) in App development. The aim of this study was to determine quality and features of smoking cessation Apps available on Android® and iOS® platforms.

METHODS

The first fifty free smoking cessation Apps available for download using the search term smoking cessation on Google Play Store and Apple App Store were chosen. Each of these Apps was analyzed and categorized based on ratings, target audience age, language, and a variety of tracking functionalities noted on the Apps. Indications and suggestions regarding either the use of NDTP or evidence-based behavior change protocols were noted.

RESULTS

There were no significant differences in the features of smoking cessation Apps on Android and iOS. Only 15 percent of all Apps analyzed on both platforms indicated some involvement of NDTP and there was no difference between the two platforms. More than 50 percent of Apps studied were downloaded over half a million times and the average user rating was 3.89/5.00 for Android and 3.72/5.00 for iOS with no significant difference.

CONCLUSIONS

Most smoking cessation Apps in both platforms offer basic tracking functionalities with limited motivational tips. Only a handful of Apps have moved beyond this role and while their development is applaudable much innovation remains.

SUITABILITY OF CURRENT EVALUATION FRAMEWORKS FOR USE IN THE HEALTH TECHNOLOGY ASSESSMENT OF MOBILE MEDICAL APPLICATIONS: A SYSTEMATIC REVIEW

OBJECTIVES

To identify and appraise existing evaluation frameworks for mobile medical applications (MMA) and determine their suitability for use in health technology assessment (HTA) of these technologies.

METHODS

Systematic searches were conducted of seven bibliographic databases to identify literature published between 2008 and 2016 on MMA evaluation frameworks. Frameworks were eligible if they were used to evaluate at least one of the HTA domains of effectiveness, safety, and/or cost and cost-effectiveness of an MMA. After inclusion, the frameworks were reviewed to determine the number and extent to which other elements of an HTA were addressed by the framework.

RESULTS

A total of forty-five frameworks were identified that assessed MMAs. All frameworks assessed whether the app was effective. Of the thirty-four frameworks that examined safety, only seven overtly evaluated potential harms from the MMA (e.g., the impact of inaccurate information). Only one framework explicitly considered a comparator. Technology specific domains were sporadically addressed.

CONCLUSION

None of the evaluation frameworks could be used, unaltered, to guide the HTA of MMAs. To use these frameworks in HTA they would need to identify relevant comparators, improve assessments of harms and consider the ongoing effect of software updates on the safety and effectiveness of MMAs. Attention should also be paid to ethical issues, such as data privacy, and technology specific characteristics.

IMPLICATIONS

Existing MMA evaluation frameworks are not suitable for use in HTA. Further research is needed before an MMA evaluation framework can be developed that will adequately inform policy makers.

A Face-Aging Smoking Prevention/Cessation Intervention for Nursery School Students in Germany: An Appearance-Focused Interventional Study

The Education Against Tobacco (EAT) network delivers smoking prevention advice in secondary schools, typically using the mirroring approach (i.e., a “selfie” altered with a face-aging app and shared with a class). In November 2017, however, the German assembly of EAT opted to expand its remit to include nursing students. To assess the transferability of the existing approach, we implemented it with the self-developed face-aging app “Smokerface” (=mixed − methods approach) in six nursing schools. Anonymous questionnaires were used to assess the perceptions of 197 students (age 18–40 years; 83.8% female; 26.4% smokers; 23.3% daily smokers) collecting qualitative and quantitative data for our cross-sectional study. Most students perceived the intervention to be fun (73.3%), but a minority disagreed that their own animated selfie (25.9%) or the reaction of their peers (29.5%) had motivated them to stop smoking. The impact on motivation not to smoke was considerably lower than experienced with seventh graders (63.2% vs. 42.0%; notably, more smokers also disagreed (45.1%) than agreed (23.5%) with this statement. Agreement rates on the motivation not to smoke item were higher in females than in males and in year 2–3 than in year 1 students. Potential improvements included greater focus on pathology (29%) and discussing external factors (26%). Overall, the intervention seemed to be appealing for nursing students.

The Emerging Imperative for a Consensus Approach Toward the Rating and Clinical Recommendation of Mental Health Apps

With over 10,000 mental health- and psychiatry-related smartphone apps available today and expanding, there is a need for reliable and valid evaluation of these digital tools. However, the updating and nonstatic nature of smartphone apps, expanding privacy concerns, varying degrees of usability, and evolving interoperability standards, among other factors, present serious challenges for app evaluation. In this article, we provide a narrative review of various schemes toward app evaluations, including commercial app store metrics, government initiatives, patient-centric approaches, point-based scoring, academic platforms, and expert review systems. We demonstrate that these different approaches toward app evaluation each offer unique benefits but often do not agree to each other and produce varied conclusions as to which apps are useful or not. Although there are no simple solutions, we briefly introduce a new initiative that aims to unify the current controversies in app elevation called CHART (Collaborative Health App Rating Teams), which will be further discussed in a second article in this series.

Facial-Aging Mobile Apps for Smoking Prevention in Secondary Schools in Brazil: Appearance-Focused Interventional Study

Background

Most smokers start smoking during their early adolescence, often with the idea that smoking is glamorous. Interventions that harness the broad availability of mobile phones as well as adolescents' interest in their appearance may be a novel way to improve school-based prevention. A recent study conducted in Germany showed promising results. However, the transfer to other cultural contexts, effects on different genders, and implementability remains unknown.

Objective

In this observational study, we aimed to test the perception and implementability of facial-aging apps to prevent smoking in secondary schools in Brazil in accordance with the theory of planned behavior and with respect to different genders.

Methods

We used a free facial-aging mobile phone app (“Smokerface”) in three Brazilian secondary schools via a novel method called mirroring. The students’ altered three-dimensional selfies on mobile phones or tablets and images were “mirrored” via a projector in front of their whole grade. Using an anonymous questionnaire, we then measured on a 5-point Likert scale the perceptions of the intervention among 306 Brazilian secondary school students of both genders in the seventh grade (average age 12.97 years). A second questionnaire captured perceptions of medical students who conducted the intervention and its conduction per protocol.

Results

The majority of students perceived the intervention as fun (304/306, 99.3%), claimed the intervention motivated them not to smoke (289/306, 94.4%), and stated that they learned new benefits of not smoking (300/306, 98.0%). Only a minority of students disagreed or fully disagreed that they learned new benefits of nonsmoking (4/306, 1.3%) or that they themselves were motivated not to smoke (5/306, 1.6%). All of the protocol was delivered by volunteer medical students.

Conclusions

Our data indicate the potential for facial-aging interventions to reduce smoking prevalence in Brazilian secondary schools in accordance with the theory of planned behavior. Volunteer medical students enjoyed the intervention and are capable of complete implementation per protocol.

Privacy Issues in Smartphone Applications: An Analysis of Headache/Migraine Applications

BACKGROUND

Headache diaries are a mainstay of migraine management. While many commercial smartphone applications (apps) have been developed for people with migraine, little is known about how well these apps protect patient information and whether they are secure to use.

OBJECTIVE

We sought to assess whether there are privacy issues surrounding apps so that physicians and patients could better understand what medical information patients are providing to the app companies, and the potential privacy implications of how the app companies (and other third parties) might use that information.

METHODS

We conducted a systematic search of the most popular "headache" and "migraine" apps and developed a database of the types of data the apps requested for input by the user and whether the apps had clear privacy policies. We also examined the content of the privacy policies.

RESULTS

Twenty-nine apps were examined (14 diary apps, 15 relaxation apps). Of the diary applications, 79% (11/14) had visible privacy policies. Of the diary apps with privacy policies, all (11/11) stated whether or not the app collects and stores information remotely. A total of 55% (6/11) stated that some user data were used to serve targeted advertisements. A total of 11/15 (73%) of the relaxation apps had privacy policies.

CONCLUSIONS

Headache apps shared information with third parties, posing privacy risks partly because there are few legal protections against the sale or disclosure of data from medical apps to third parties.

Creating boundaries to empower digital health technology

SummaryThe potential of digital health tools such as smartphones and sensors to increase access to and enhance delivery of healthcare is well known. However, a lack of regulation and delineation between those technologies seeking to offer direct clinical diagnostics and treatments and those involving clinical care enhancements or direct-to-consumer resources has led to patient and clinician confusion about the appropriate use and role of digital health. Here, we propose that creating boundaries and better defining the scope of digital health technology will advance the field through matching the right use cases with the right tools. We further propose that ethical clinicians, as stewards of standard of care, are well suited to uphold these boundaries and to safeguard best practices in digital health.Declaration of interestH.H. is an employee of Verily Life Sciences and owns equity in this company. The views expressed here are those of the authors and are not official views of Verily Life Sciences.

A Cross-Sectional Study of Prominent US Mobile Health Applications: Evaluating the Current Landscape

Mobile health (mHealth) could offer unprecedented opportunity to provide medical support closer to the users. We have selected some relevant criteria to describe 100 apps from Google Play store and Apple's App Store's top suggestions in medical category. These characteristics were compared based on the paid or free nature of the apps, the target users: consumers or healthcare professionals, and the platform: Android or iOS. Seventeen provided functionalities and 27 medical subjects covered by these apps were also extracted. Our study shows that even in top rated mHealth apps, a high proportion lacks some basic criteria regarding the quality of the apps including the presence of a privacy policy, describing content sources, participation of the target users in the app development, etc. Paid apps did not ensure better quality compared to free apps. The current mHealth market is not mature enough to be used widely and recommended by healthcare professionals.

Methods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps

Commercial mobile apps for health behavior change are flourishing in the marketplace, but little evidence exists to support their use. This paper summarizes methods for evaluating the content, usability, and efficacy of commercially available health apps. Content analyses can be used to compare app features with clinical guidelines, evidence-based protocols, and behavior change techniques. Usability testing can establish how well an app functions and serves its intended purpose for a target population. Observational studies can explore the association between use and clinical and behavioral outcomes. Finally, efficacy testing can establish whether a commercial app impacts an outcome of interest via a variety of study designs, including randomized trials, multiphase optimization studies, and N-of-1 studies. Evidence in all these forms would increase adoption of commercial apps in clinical practice, inform the development of the next generation of apps, and ultimately increase the impact of commercial apps.

Parent Education is Changing: A Review of Smartphone Apps

PURPOSE

The purpose was to critique existing parenting apps using established criteria and health literacy guidelines.

STUDY DESIGN

Descriptive methodology was used.

METHODS

The Apple App Store was searched using the terms parenting, child health, and infant health. To be included, the apps had to have relevant content (parenting, child health, or infant health), be in English, and contain parent education. After eliminating apps that failed to meet inclusion criteria from the original 203 apps, 46 apps were reviewed. The Patient Education Materials Assessment Tool was used to evaluate the health literacy subscales called Understandability and Actionability. Content analysis included Authority, Objectivity, Accuracy, Timeliness, and Usability.

RESULTS

The majority of the apps (70%) were in English only. The price ranged from free to $4.99. The purpose, target audience, and topics varied. Although all included apps were for parents, some were for more targeted groups of parents. The source of the information was not presented in 26% of the apps. Most apps took the user to a Web site or an article to read. Functionality of the apps was limited, with none of them providing a customized experience.

CLINICAL IMPLICATIONS

Much development and research is needed before mobile health (mHealth) solutions can be recommended by nurses caring for new parents. It is critical that consumers and interdisciplinary professionals be involved in the early design phase of the product to ensure that the end product is acceptable and usable and that it will lead to healthy behaviors.