Quality Principles of App Description Texts and Their Significance in Deciding to Use Health Apps as Assessed by Medical Students: Survey Study

Digital health applications from a government-regulated directory of reimbursable health apps in Germany-a systematic review for evidence and bias

Background

The Digital Healthcare Act, passed in November 2019, authorizes healthcare providers in Germany to prescribe digital health applications (DiGA) to patients covered by statutory health insurance. If DiGA meet specific efficacy requirements, they may be listed in a special directory maintained by the German Federal Institute for Drugs and Medical Devices. Due to the lack of well-founded app evaluation tools, the objectives were to assess (I) the evidence quality situation for DiGA in the literature and (II) how DiGA manufacturers deal with this issue, as reflected by the apps available in the aforementioned directory.

Methods

A systematic review of the literature on DiGA using PubMed, Scopus, and Web of Science was started on February 4, 2023. Papers addressing the evidence for applications listed in the directory were included, while duplicates and mere study protocols not reporting on data were removed. The remaining publications were used to assess the quality of the evidence or potential gaps in this regard. Results were aggregated in tabular form.

Results

The review identified fourteen relevant publications. Six studies suggested inadequate scientific evidence, five mentioned shortcomings of tools for validating DiGA-related evidence, and four publications described a high potential for bias, potentially influencing the validity of the results. Concerns about limited external generalizability were also raised.

Conclusions

The literature review found evidence-related gaps that must be addressed with adequate measures. Our findings can serve as a basis for a plea for a more detailed examination of the quality of evidence in the DiGA context.

Orthopaedic and trauma surgeons' prioritisation of app quality principles based on their demographic background

BACKGROUND

Although apps are becoming increasingly relevant in healthcare, there is limited knowledge about how healthcare professionals perceive "quality" in this context and how quality principles that can aid them in assessing health-related apps may be prioritised. The objective was to investigate physicians' views of predefined (general) quality principles for health apps and to determine whether a ranking algorithm applied to the acquired data can provide stable results against various demographic influences and may thus be appropriate for prioritisation.

METHODS

Participants of an online survey of members of two German professional orthopaedics associations conducted between 02/12/2019 and 02/01/2020 were asked about their perception of a set of quality principles for health apps (i.e., "practicality," "risk adequacy," "ethical soundness," "legal conformity," "content validity," "technical adequacy," "usability," "resource efficiency," and "transparency"). Structured as a Kano survey, for each principle, there were questions about its perceived relevance and opinions regarding the presence or absence of corresponding characteristics. The available data were evaluated descriptively, and a newly developed method for prioritisation of the principles was applied overall and to different demographic strata (for validation).

RESULTS

Three hundred eighty-two datasets from 9503 participants were evaluated. Legal conformity, content validity, and risk adequacy filled ranks one to three, followed by practicability, ethical soundness, and usability (ranks 4 to 6). Technical adequacy, transparency, and resource efficiency ranked last (ranks 7 to 9). The ranking based on the proposed method was relatively stable, irrespective of demographic factors. The principles were seen as essential, with one exception ("resource efficiency"). Only those with little to no interest in digitisation (22/382, 5.8%) rated the nine principles indifferently.

CONCLUSIONS

The specified quality principles and their prioritisation can lay a foundation for future assessments of apps in the medical field. Professional societies build upon this to highlight opportunities for digital transformations in medicine and encourage their members to participate.

Proposal of a New Rating Concept for Digital Health Applications in Orthopedics and Traumatology

BACKGROUND

Health-related mobile applications (apps) are rapidly increasing in number. There is an urgent need for assessment tools and algorithms that allow the usability and content criteria of these applications to be objectively assessed. The aim of this work was to establish and validate a concept for orthopedic societies to rate health apps to set a quality standard for their safe use.

METHODS

An objective rating concept was created, consisting of nine quality criteria. A self-declaration sheet for app manufacturers was designed. Manufacturers completed the self-declaration, and the app was examined by independent internal reviewers. The pilot validation and analysis were performed on two independent health applications. An algorithm for orthopedic societies was created based on the experiences in this study flow.

RESULTS

"Sprunggelenks-App" was approved by the reviewers with 45 (98%) fulfilled criteria and one (2%) unfulfilled criterion. "Therapie-App" was approved, with 28 (61%) met criteria, 6 (13%) unfulfilled criteria and 12 (26%) criteria that could not be assessed. The self-declaration completed by the app manufacturer is recommended, followed by a legal and technical rating performed by an external institution. When rated positive, the societies' internal review using independent raters can be performed. In case of a positive rating, a visual certification can be granted to the manufacturer for a certain time frame.

CONCLUSION

An objective rating algorithm is proposed for the assessment of digital health applications. This can help societies to improve the quality assessment, quality assurance and patient safety of those apps. The proposed concept must be further validated for inter-rater consistency and reliability.

A Comparison of Functional Features of Chinese and US Mobile Apps for Pregnancy and Postnatal Care: A Systematic App Store Search and Content Analysis

Background

Pregnancy to postpartum (PtP) applications (apps) are becoming more common tools to document everything from pregnancy and delivery to nutrient allocation, life taboos, and infant medical examinations. However, the dependability, quality, and efficacy of these apps remain unclear. This study examined the features and functions of mobile PtP care apps accessible in China and the United States and to identify the major gaps that need to be addressed.

Methods

Apps were selected by searching the Apple App Store and Android Markets (in the US and China) for the terms “pregnancy” and “postpartum” in Chinese and English. The apps' security, quality, and effectiveness were investigated, and chi-square tests and analysis of variance were performed to examine the differences in characteristics between apps available in the US and China.

Results

A total of 84 mobile PtP care apps (45 from the US and 39 from China) were included. A total of 89.7% (35/39) of Chinese mobile apps did not provide safety statements or supporting evidence. The objective app quality ratings for Chinese and US apps were 3.20 ± 0.48 (mean ± standard deviation) and 3.56 ± 0.45, respectively (p > 0.05). A greater number of Chinese apps provided app-based monitoring functions, namely recording fetal size (n = 18, 46.2% in China vs. n = 3, 6.7% in the US), contractions (n = 11, 28.2% in China vs. n = 0, 0% in the US), pregnancy weight (n = 11, 28.2% in China vs. 0, 0% in the US), and pregnancy check-up reminders (n = 10, 25.6% in China vs. n = 0, 0% in the US). Meanwhile, a greater number of US apps provided exercise modules, namely pregnancy yoga (n = 2, 5.1% in China vs. n = 21, 46.7% in the US), pregnancy workouts (n = 2, 5.1% in China vs. n = 13, 28.9% in the US), and pregnancy meditation (n = 0, 0% in China vs. 10, 22.2% in the US) (p < 0.01). A medium security risk was identified for 40% (18/45) of apps in the US and 82.1% (32/39) of apps in China (p < 0.01).

Conclusions

The functionality and characteristics of in-store mobile apps for PtP care varied between China and the US. Both countries' apps, particularly Chinese apps, encountered issues related to a lack of evidence-based information, acceptable content risk, and program evaluations. Both countries' apps lacked proper mental health care functions. The findings suggest that the design of app features should be enhanced in both countries, and increased interaction between app creators and users is recommended.

Prioritization of Quality Principles for Health Apps Using the Kano Model: Survey Study

Background

Health apps are often used without adequately taking aspects related to their quality under consideration. This may partially be due to inadequate awareness about necessary criteria and how to prioritize them when evaluating an app.

Objective

The aim of this study was to introduce a method for prioritizing quality attributes in the mobile health context. To this end, physicians were asked about their assessment of nine app quality principles relevant in health contexts and their responses were used as a basis for designing a method for app prioritization. Ultimately, the goal was to aid in making better use of limited resources (eg, time) by assisting with the decision as to the specific quality principles that deserve priority in everyday medical practice and those that can be given lower priority, even in cases where the overall principles are rated similarly.

Methods

A total of 9503 members of two German professional societies in the field of orthopedics were invited by email to participate in an anonymous online survey over a 1-month period. Participants were asked to rate a set of nine app quality principles using a Kano survey with functional and dysfunctional (ie, positively and negatively worded) questions. The evaluation was based on the work of Kano (baseline), supplemented by a self-designed approach.

Results

Among the 9503 invited members, 382 completed relevant parts of the survey (return rate of 4.02%). These participants were equally and randomly assigned to two groups (test group and validation group, n=191 each). Demographic characteristics did not significantly differ between groups (all P>.05). Participants were predominantly male (328/382, 85.9%) and older than 40 years (290/382, 75.9%). Given similar ratings, common evaluation strategies for Kano surveys did not allow for conclusive prioritization of the principles, and the same was true when using the more elaborate approach of satisfaction and dissatisfaction indices following the work of Timko. Therefore, an extended, so-called “in-line-of-sight” method was developed and applied for this evaluation. Modified from the Timko method, this approach is based on a “point of view” (POV) metric, which generates a ranking coefficient. Although the principles were previously almost exclusively rated as must-be (with the exception of resource efficiency), which was not conducive to their prioritization, the new method applied from the must-be POV resulted in identical rankings for the test and validation groups: (1) legal conformity, (2) content validity, (3) risk adequacy, (4) practicality, (5) ethical soundness, (6) usability, (7) transparency, (8) technical adequacy, and (9) resource efficiency.

Conclusions

Established survey methodologies based on the work of Kano predominantly seek to categorize the attributes to be evaluated. The methodology presented here is an interesting option for prioritization, and enables focusing on the most important criteria, thus saving valuable time when reviewing apps for use in the medical field, even with otherwise largely similar categorization results. The extent to which this approach is applicable beyond the scenario presented herein requires further investigation.

[Identification of rheumatological health apps in the Apple app store applying the "semiautomatic retrospective app store analysis" method : A longitudinal observation]

Hintergrund

Die App Stores von Apple und Google bieten eine Vielzahl von Gesundheits-Apps an. Das Auffinden qualitativ hochwertiger Apps ist immer noch eine Herausforderung.

Fragestellung

Lassen sich unter Anwendung der SARASA(„semiautomated retrospective App Store analysis“)-Methode für das Fachgebiet Rheumatologie deutschsprachige Apps identifizieren?

Material und Methode

SARASA ist eine Methode zur teilautomatisierten Auswahl und Charakterisierung von App Store-gelisteten Apps nach formalen Kriterien. Nach der ersten Anwendung in 02/2018 wurde SARASA 02/2020 erneut auf den Apple App Store angewendet.

Ergebnisse

In 02/2018 konnten für Apps in den Store-Kategorien „Medizin“ oder „Gesundheit und Fitness“ Metadaten zu 103.046 Apps und bei einer erneuten Erhebung in 02/2020 Daten zu 94.735 Apps über das deutsche Frontend des Apple App Stores ausgelesen werden. Im Jahr 2018 wurden nach Anwendung der Suchbegriffe 59 Apps mit einer deutschsprachigen Beschreibung für das Fachgebiet Rheumatologie identifiziert, 2020 waren dies 53 Apps, die jeweils manuell weiter überprüft wurden; 2018 waren noch mehr der gefundenen Apps für Patienten als für Ärzte vorgesehen, dies war 2020 ausgeglichener. Zudem zeigte sich, dass bei bestimmten Krankheitsbildern von den App-Entwicklern keine Bearbeitungen erfolgten. Die prozentuale Verteilung von Treffern nach Suchbegriffen zeigte im Vergleich von 2018 zu 2020 große Schwankungen.

Diskussion

Die SARASA-Methode stellt ein hilfreiches Werkzeug dar, um Gesundheits-Apps teilautomatisiert zu identifizieren, die vordefinierten, formalen Kriterien entsprechen. Die inhaltliche Qualität muss anschließend manuell überprüft werden. Weiterentwicklungen der SARASA-Methode und die weitere Konsentierung und Standardisierung von Qualitätskriterien sind sinnvoll. Qualitätskriterien sollten beim Angebot von Gesundheits-Apps in den App-Stores berücksichtigt werden.

Digital Health Usage and Awareness among MedicalStudents: A Survey Study

   BACKGROUND: In the era of increasingly expanding digital ecosystem, health misinformation became highly risky, especially the information sources which are non-evidence based. This problem is magnified due to the vulnerability of most internet users, especially the adolescents information seekers who lack health literacy. AIM: The aim of this study is to explore the pattern of digital health usage among medical university students and their level of awareness towards choosing online health information (OHI). METHODS: A cross-sectional study of 480 medical students from 1st, 2nd, and 3rd academic years, at the Faculty of Medicine, Helwan University, between October 2019 and October 2020. Students took an online structured questionnaire on google forms. RESULTS: Mobile smartphones were the most widely used technological devices by participants. Almost all participants used Internet daily 98.8%, with the highest preference to social media 72.5% and text messaging 74.8%. Seeking health information was a common practice for personal health-related purpose of using Internet 87.1%, despite the lack in checking the reliability of messages and quality of provider. Knowledge about advantages and disadvantages of digital tools was also deficient among participant medical students. CONCLUSION: Despite the high technology adoption among the university medical students, they lack the needed knowledge and skills for proper search, choosing, and evaluation of OHI.

Influences on the Uptake of Health and Well-being Apps and Curated App Portals: Think-Aloud and Interview Study

BACKGROUND

Health and well-being smartphone apps can provide a cost-effective solution to addressing unhealthy behaviors. The selection of these apps tends to occur in commercial app stores, where thousands of health apps are available. Their uptake is often influenced by popularity indicators. However, these indicators are not necessarily associated with app effectiveness or evidence-based content. Alternative routes to app selection are increasingly available, such as via curated app portals, but little is known about people's experiences of them.

OBJECTIVE

The aim of this study is to explore how people select health apps on the internet and their views on curated app portals.

METHODS

A total of 18 UK-based adults were recruited through social media and asked during an in-person meeting to verbalize their thoughts while searching for a health or well-being app on the internet on a platform of their choice. The search was then repeated on 2 curated health app portals: the National Health Service Apps Library and the Public Health England One You App portal. This was followed by semistructured interviews. Data were analyzed using framework analysis, informed by the Capability, Opportunity, Motivation-Behavior model and the Theoretical Domains Framework.

RESULTS

Searching for health and well-being apps on the internet was described as a minefield. App uptake appeared to be influenced by participants' capabilities such as app literacy skills and health and app awareness, and opportunities including the availability of apps, app esthetics, the price of an app, and social influences. Motivation factors that seemed to affect the uptake were perceived competence, time efficiency, perceived utility and accuracy of an app, transparency about data protection, commitment and social identity, and a wide range of emotions. Social influences and the perceived utility of an app were highlighted as particularly important. Participants were not previously aware of curated portals but found the concept appealing. Curated health app portals appeared to engender trust and alleviate data protection concerns. Although apps listed on these were perceived as more trustworthy, their presentation was considered disappointing. This disappointment seemed to stem from the functionality of the portals, lack of user guidance, and lack of tailored content to an individual's needs.

CONCLUSIONS

The uptake of health and well-being apps appears to be primarily affected by social influences and the perceived utility of an app. App uptake via curated health app portals perceived as credible may mitigate concerns related to data protection and accuracy, but their implementation must better meet user needs and expectations.

A Possible Mobile Health Solution in Orthopedics and Trauma Surgery: Development Protocol and User Evaluation of the Ankle Joint App

Background

Ankle sprains are one of the most frequent sports injuries. With respect to the high prevalence of ankle ligament injuries and patients’ young age, optimizing treatment and rehabilitation is mandatory to prevent future complications such as chronic ankle instability or osteoarthritis.

Objective

In modern times, an increasing amount of smartphone usage in patient care is evident. Studies investigating mobile health (mHealth)–based rehabilitation programs after ankle sprains are rare. The aim of this study was to expose any issues present in the development process of a medical app as well as associated risks and chances.

Methods

The development process of the Ankle Joint App was defined in chronological order using a protocol. The app’s quality was evaluated using the (user) German Mobile App Rating Scale (MARS-G) by voluntary foot and ankle surgeons (n=20) and voluntary athletes (n=20).

Results

A multidisciplinary development team built a hybrid app with a corresponding backend structure. The app’s content provides actual medical literature, training videos, and a log function. Excellent interrater reliability (interrater reliability=0.92; 95% CI 0.86-0.96) was obtained. The mean overall score for the Ankle Joint App was 4.4 (SD 0.5). The mean subjective quality scores were 3.6 (surgeons: SD 0.7) and 3.8 (athletes: SD 0.5). Behavioral change had mean scores of 4.1 (surgeons: SD 0.7) and 4.3 (athletes: SD 0.7). The medical gain value, rated by the surgeons only, was 3.9 (SD 0.6).

Conclusions

The data obtained demonstrate that mHealth-based rehabilitation programs might be a useful tool for patient education and collection of personal data. The achieved (user) MARS-G scores support a high quality of the tested app. Medical app development with an a priori defined target group and a precisely intended purpose, in a multidisciplinary team, is highly promising. Follow-up studies are required to obtain funded evidence for the ankle joints app’s effects on economical and medical aspects in comparison with established nondigital therapy paths.

Assessment of Mobile Health Apps Using Built-In Smartphone Sensors for Diagnosis and Treatment: Systematic Survey of Apps Listed in International Curated Health App Libraries

Background

More than a million health and well-being apps are available from the Apple and Google app stores. Some apps use built-in mobile phone sensors to generate health data. Clinicians and patients can find information regarding safe and effective mobile health (mHealth) apps in third party–curated mHealth app libraries.

Objective

These independent Web-based repositories guide app selection from trusted lists, but do they offer apps using ubiquitous, low-cost smartphone sensors to improve health? This study aimed to identify the types of built-in mobile phone sensors used in apps listed on curated health app libraries, the range of health conditions these apps address, and the cross-platform availability of the apps.

Methods

This systematic survey reviewed three such repositories (National Health Service Apps Library, AppScript, and MyHealthApps), assessing the availability of apps using built-in mobile phone sensors for the diagnosis or treatment of health conditions.

Results

A total of 18 such apps were identified and included in this survey, representing 1.1% (8/699) to 3% (2/76) of all apps offered by the respective libraries examined. About one-third (7/18, 39%) of the identified apps offered cross-platform Apple and Android versions, with a further 50% (9/18) only dedicated to Apple and 11% (2/18) to Android. About one-fourth (4/18, 22%) of the identified apps offered dedicated diagnostic functions, with a majority featuring therapeutic (9/18, 50%) or combined functionality (5/18, 28%). Cameras, touch screens, and microphones were the most frequently used built-in sensors. Health concerns addressed by these apps included respiratory, dermatological, neurological, and anxiety conditions.

Conclusions

Diligent mHealth app library curation, medical device regulation constraints, and cross-platform differences in mobile phone sensor architectures may all contribute to the observed limited availability of mHealth apps using built-in phone sensors in curated mHealth app libraries. However, more efforts are needed to increase the number of such apps on curated lists, as they offer easily accessible low-cost options to assist people in managing clinical conditions.

A design and evaluation framework for digital health interventions

Digital health interventions (DHIs) have the potential to help the growing number of chronic disease patients better manage their everyday lives. However, guidelines for the systematic development of DHIs are still scarce. The current work has, therefore, the objective to propose a framework for the design and evaluation of DHIs (DEDHI). The DEDHI framework is meant to support both researchers and practitioners alike from early conceptual DHI models to large-scale implementations of DHIs in the healthcare market.

Quality Awareness and Its Influence on the Evaluation of App Meta-Information by Physicians: Validation Study

Background

Meta-information provided about health apps on app stores is often the only readily available source of quality-related information before installation.

Objective

The purpose of this study was to assess whether physicians deem a predefined set of quality principles as relevant for health apps; whether they are able to identify corresponding information in a given sample of app descriptions; and whether, and how, this facilitates their informed usage decisions.

Methods

All members of the German Society for Internal Medicine were invited by email to participate in an anonymous online survey over a 6-week period. Participants were randomly assigned one app description focusing on cardiology or pulmonology. In the survey, participants were asked three times about whether the assigned description sufficed for a usage decision: they were asked (1) after giving an appraisal of the relevance of nine predefined app quality principles, (2) after determining whether the descriptions covered the quality principles, and (3) after they assessed the availability of detailed quality information by means of 25 additional key questions. Tests for significance of changes in their decisions between assessments 1 and 2, and between assessments 2 and 3, were conducted with the McNemar-Bowker test of symmetry. The effect size represents the discordant proportion ratio sum as a quotient of the test statistics of the Bowker test and the number of observation units. The significance level was set to alpha=.05 with a power of 1-beta=.95.

Results

A total of 441 of 724 participants (60.9%) who started the survey fully completed the questionnaires and were included in the evaluation. The participants predominantly rated the specified nine quality principles as important for their decision (approximately 80%-99% of ratings). However, apart from the practicality criterion, information provided in the app descriptions was lacking for both groups (approximately 51%-92%). Reassessment of the apps led to more critical assessments among both groups. After having familiarized themselves with the nine quality principles, approximately one-third of the participants (group A: 63/220, 28.6%; group B: 62/221, 28.1%) came to more critical usage decisions in a statistically significant manner (McNemar-Bowker test, groups A and B: P<.001). After a subsequent reassessment with 25 key questions, critical appraisals further increased, although not in a statistically significant manner (McNemar-Bowker, group A: P=.13; group B: P=.05).

Conclusions

Sensitizing physicians to the topic of quality principles via questions about attitudes toward established quality principles, and letting them apply these principles to app descriptions, lead to more critical appraisals of the sufficiency of the information they provided. Even working with only nine generic criteria was sufficient to bring about the majority of decision changes. This may lay the foundation for aiding physicians in their app-related decision processes, without unduly taking up their valuable time.