Assessing usability of eHealth technology: A comparison of usability benchmarking instruments

Assessing the usability of Accessercise to increase physical activity in adults with physical disabilities: A qualitative think-aloud study

BACKGROUND

Mobile health (mHealth) applications (apps) offer a convenient way to increase physical activity to people with disabilities. While several mHealth apps have been developed for this population, there is limited evidence assessing app usability and how this might impact physical activity.

OBJECTIVE

To investigate the usability of a novel mHealth app, Accessercise, that aims to increase physical activity in people with physical disabilities.

METHODS

Twelve adults with different physical disabilities participated in a face-to-face think-aloud interview. Interviews were analysed using deductive content analysis based on the User Version of the Mobile Application Rating Scale (uMARS).

FINDINGS

Data mapped onto 12 uMARS domains; most participants expressed positive views concerning Accessercise, namely, entertainment, customisation, tailoring to the target group, ease of use and navigation, and visual information. Some additional elements were viewed positively but required modification to improve usability, perceived credibility, and relevance, including the app's layout, visual appeal, interactivity, and quality/quantity of information.

CONCLUSION

This study provides an example of how the think-aloud method can be employed to evaluate mHealth apps that aims to increase physical activity in people with physical disabilities. Therefore, developers and researchers could use this study to inform future usability evaluations in this area.

Comparison of usability and user-friendliness of three FeNO analyzers in a general population cohort of the LEAD study

Exhaled nitric oxide (FeNO) is a marker for airway inflammation measured by hand-held or stationary analyzers, but their usability was not previously assessed. NIOX VERO (CN), NObreath (BN), Vivatmo pro (BV), and CLD88 analyzer (reference, EC) were compared in a prospective study of the general population LEAD (Lung, hEart, sociAl, boDy) cohort, including the System Usability Scale and tests for equivalence at a clinically relevant range of ≤ 70 ppb with linear models and Bland-Altman plots. In 486 participants (62.4 ± 14.2 years old, 48.1% female), all hand-held analyzers had a good usability score, with BN scoring best. BV required the fewest attempts and time to measurement success, followed by BN. The FeNO results were clinically equivalent between devices (difference to EC 0.7-7.5 ppb) with increasing variability at higher FeNO values. The analyzers had an agreement of ≥ 95% at the threshold of ≥ 40 ppb. CN showed the lowest difference to EC, followed by BV. All portable analyzers showed good usability with an above-average usability score. The best usability score was observed with the BN device, while the BV device had the shortest measuring time and the fewest additional attempts. The lowest difference to the stationary EC analyzers was observed with the CN device.

Patients' Experienced Usability and Satisfaction With Digital Health Solutions in a Home Setting: Instrument Validation Study

Background

The field of digital health solutions (DHS) has grown tremendously over the past years. DHS include tools for self-management, which support individuals to take charge of their own health. The usability of DHS, as experienced by patients, is pivotal to adoption. However, well-known questionnaires that evaluate usability and satisfaction use complex terminology derived from human-computer interaction and are therefore not well suited to assess experienced usability of patients using DHS in a home setting.

Objective

This study aimed to develop, validate, and assess an instrument that measures experienced usability and satisfaction of patients using DHS in a home setting.

Methods

The development of the "Experienced Usability and Satisfaction with Self-monitoring in the Home Setting" (GEMS) questionnaire followed several steps. Step I consisted of assessing the content validity, by conducting a literature review on current usability and satisfaction questionnaires, collecting statements and discussing these in an expert meeting, and translating each statement and adjusting it to the language level of the general population. This phase resulted in a draft version of the GEMS. Step II comprised assessing its face validity by pilot testing with Amsterdam University Medical Center's patient panel. In step III, psychometric analysis was conducted and the GEMS was assessed for reliability.

Results

A total of 14 items were included for psychometric analysis and resulted in 4 reliable scales: convenience of use, perceived value, efficiency of use, and satisfaction.

Conclusions

Overall, the GEMS questionnaire demonstrated its reliability and validity in assessing experienced usability and satisfaction of DHS in a home setting. Further refinement of the instrument is necessary to confirm its applicability in other patient populations in order to promote the development of a steering mechanism that can be applied longitudinally throughout implementation, and can be used as a benchmarking instrument.

Protocol for a usability and pilot implementation study of a digital medical device to assess pain in non-verbal people with dementia in Portuguese residential care facilities

Objective

People living with moderate to severe dementia (PLWD) are often unable to self-report pain. This matter is of particular concern given that up to 80% experience chronic pain. Mistreated or untreated pain in PLWD is associated with symptoms such as agitation and aggression, and unnecessary use of antipsychotic agents. Further, it can also engender mental burden in formal caregivers. The PainChek® App, a regulatory cleared class I medical device, enables the assessment and monitoring of pain in people who cannot verbalise it, such as those with moderate to severe dementia. To date there are no data on the real-world use of the PainChek® App in Portugal. To address this gap, we report the protocol of a pilot study, which combines usability evaluation and implementation research.

Methods

Usability evaluation of the PainChek® (Portuguese) App will be guided by the ISO framework, focused on effectiveness, efficiency and user satisfaction. Implementation research will combine qualitative interviews to inform the implementation process, a longitudinal study of formal caregivers' psychological variables, implementation outcomes, plus qualitative interviews to explore the 'hows' and 'whys' of implementation. The NASSS framework will be used as an implementation framework, together with the COM-B model and the Theoretical Domains Framework.

Results

The usability and implementation studies have received ethics approval from the Egas Moniz Ethics Committee, under numbers 1367 and 64/24, respectively.

Conclusion

This study is expected to inform the scale-up of the PainChek® (Portuguese) App in real-world settings and establish a foundation for a larger effectiveness and implementation study.

Using FRAME Documentation to Achieve RE-AIM Goals During Iterative, Stakeholder-Engaged Refinement of a Family Management Intervention for Parents of Preterm Infants

AIMS

To co-identify adaptations with key stakeholders needed to optimise elements of a video-based intervention (i.e., PREEMIE PROGRESS [PP]), which trains parents in evidence-based family management skills to care for their very preterm infant in the neonatal intensive care unit (NICU).

DESIGN

Descriptive qualitative study oriented with a pragmatic philosophy, informed by the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework and the framework for reporting adaptations and modifications-expanded (FRAME).

METHODS

Semistructured interviews to identify potential adaptations with key stakeholders: family management researchers (n = 5), clinicians (n = 9), technology experts (n = 5) and parents of preterm infants (n = 17). Weekly design team meetings to select and implement high-priority adaptations necessary for the next research phase. Monthly NICU parent partnership meetings to review adaptations and make recommendations for potential adaptations with conflicting data.

RESULTS

Stakeholders (N = 36) suggested 98 potential adaptations: 32 (33.0%) were completed, 8 (8.2%) were abandoned, 5 (5.2%) have work that is ongoing and 52 (53.6%) were tabled for future research phases. Content adaptations (70, 71.4%) were the most frequently suggested adaptation type. Potential adaptations mostly addressed RE-AIM dimensions of effectiveness (43, 43.9%), and implementation (46, 46.9%) and were directed at the parent (i.e., intervention recipient) level (79, 81.4%).

CONCLUSION

Use of the RE-AIM framework ensured we systematically identified needed adaptations with key stakeholders across a range of dimensions that would improve PP for parents now and in future phases of this research.

IMPLICATIONS FOR THE PROFESSION

Co-identifying potential adaptations with key stakeholders, paired with FRAME documentation, can help nurses prioritise adaptations most appropriate for each phase of implementation.

IMPACT

Our paper highlights for nurse clinicians and researchers how FRAME documentation of potential adaptations can support stakeholder engagement and a systematic approach to incorporating adaptations throughout all phases of the research process, thereby shortening the evidence to practice gap.

REPORTING METHOD

COREQ guidelines for qualitative reporting.

PATIENT/PUBLIC CONTRIBUTION

The research team was supported by members of the NICU's Parent Partnership Council (PPC), whose mission is to promote family-centred care improvement projects and research within the NICU. This committee is comprised of nursing, physician, allied health leadership and parents of infants previously hospitalised in the NICU. The NICU PPC met monthly to review conflicting data on potential adaptations and provide recommendations on adaptation decisions.

Assessing the interdependency among effectiveness, satisfaction and efficient use of the Lightwave Health Information Management System (LHIMS) by health professionals in Ghana

BACKGROUND

The success of an electronic health records (EHR) system is dependent on the effectiveness, satisfaction, and efficient use of these systems by health professionals. This paper explores the interdependency among effectiveness, satisfaction, and efficient use of Lightwave Health Information Management Systems (LHIMS) which is an EHR by health professionals.

METHODS

A non-interventional descriptive cross-sectional study design was employed for this research. The study used stratified random sampling for the selection of participants. The population for the study was all the health professionals in the Central Region of Ghana. The number of health professionals who responded to the questions was 1126.

RESULTS

The findings suggest that understanding the interdependency of efficiency, effectiveness and satisfaction is crucial for the successful implementation and adoption of LHIMS in healthcare organisations. The study investigated the interdependence among effectiveness, satisfaction, and efficient use of the LHIMS by health professionals, and found a positive but weak significant relationship between these factors.

CONCLUSION

In conclusion, this study aimed to assess the interdependency among effectiveness, satisfaction, and efficient use of LHIMS by health professionals. The. results support a positive but weak significant relationship between satisfaction, effectiveness, and efficient use of the LHIMS. These findings have implications for policymakers and hospital management in their efforts to improve the use of LHIMS. Policymakers can focus on any one of the three usability areas to enhance LHIMS use, with visible effects across the other two domains.

Designing for Improved Patient Experiences in Home Dialysis: Usability and User Experience Findings From User-Based Evaluation Study With Patients With Chronic Conditions

BACKGROUND

Chronic kidney disease affects 10% of the population worldwide, and the number of patients receiving treatment for end-stage kidney disease is forecasted to increase. Therefore, there is a pressing need for innovative digital solutions that increase the efficiency of care and improve patients' quality of life. The aim of the eHealth in Home Dialysis project is to create a novel eHealth solution, called eC4Me, to facilitate predialysis and home dialysis care for patients with chronic kidney disease.

OBJECTIVE

Our study aimed to evaluate the usability, user experience (UX), and patient experience (PX) of the first version of the eC4Me solution.

METHODS

We used a user-based evaluation approach involving usability testing, questionnaire, and interview methods. The test sessions were conducted remotely with 10 patients with chronic kidney disease, 5 of whom had used the solution in their home environment before the tests, while the rest were using it for the first time. Thematic analysis was used to analyze user test and questionnaire data, and descriptive statistics were calculated for the UMUX (Usability Metric for User Experience) scores.

RESULTS

Most usability problems were related to navigation, the use of terminology, and the presentation of health-related data. Despite usability challenges, UMUX ratings of the solution were positive overall. The results showed noteworthy variation in the expected benefits and perceived effort of using the solution. From a PX perspective, it is important that the solution supports patients' own health-related goals and fits with the needs of their everyday lives with the disease.

CONCLUSIONS

A user-based evaluation is a useful and necessary part of the eHealth solution development process. Our study findings can be used to improve the usability and UX of the evaluated eC4Me solution. Patients should be actively involved in the solution development process when specifying what information is relevant for them. Traditional usability tests complemented with questionnaire and interview methods can serve as a meaningful methodological approach for gaining insight not only into usability but also into UX- and PX-related aspects of digital health solutions.

Electronic maternal and child health application usability, feasibility and acceptability among healthcare providers in Amhara region, Ethiopia

An innovative electronic Maternal and Child Health (eMCH) application was developed to support operational and clinical decision-making in maternal and child health services. End-user-based evaluation of eHealth application is a critical step to ascertain how successfully users can learn and use it, and improve the technology. Therefore, this study aimed to evaluate the eMCH tool usability, feasibility, and acceptability among healthcare providers (HCPs) in the Amhara region, Ethiopia. A cross-sectional study was conducted among HCPs working in six public healthcare facilities. The usability evaluation was done on 24 HCPs across three professional categories using the ISO 9241-11 usability guideline. One hundred nine HCPs were participated in the feasibility and acceptability study. Data were collected using a standard usability tool, think-aloud protocol, a self-administered approach, and Open Broadcaster Software Studio version 26.1.1 video recorder. Descriptive statistics were used to describe the data. A Kruskal-Wallis test was used to measure the association between mean scores and categories of HCPs. The recorded videos were used for the log file analysis method. None of the HCP categories were able to complete all the tasks without errors. The average number of errors and restarts were 7.5 and 2.8, respectively. The average number of restarts was directly proportional to the average number of errors. The participants successfully completed more than 70% of the tasks without requiring any assistance or guidance. Forty-seven comments or errors were identified from the think-aloud analysis and 22 comments from the usability metrics analysis. Overall, statistically significant performance differences were observed among the three HCP groups across the majority of the usability evaluation metrics. Fifty-seven percent of HCPs scored higher than the mean on the feasibility study. Slightly higher than half, 56 (51.4%), of the HCPs scored higher than the mean score on the acceptability study. The usability evaluation identified vital comments and usability flaws that were essential for the eMCH tool to be upgraded. The tool was feasible and acceptable as reported by end-users. Therefore, the errors and usability flaws of the tool should be fixed before deployment to other healthcare settings.

Quality assessment of mHealth apps: a scoping review

Introduction

The number of mHealth apps has increased rapidly during recent years. Literature suggests a number of problems and barriers to the adoption of mHealth apps, including issues such as validity, usability, as well as data privacy and security. Continuous quality assessment and assurance systems might help to overcome these barriers. Aim of this scoping review was to collate literature on quality assessment tools and quality assurance systems for mHealth apps, compile the components of the tools, and derive overarching quality dimensions, which are potentially relevant for the continuous quality assessment of mHealth apps.

Methods

Literature searches were performed in Medline, EMBASE and PsycInfo. Articles in English or German language were included if they contained information on development, application, or validation of generic concepts of quality assessment or quality assurance of mHealth apps. Screening and extraction were carried out by two researchers independently. Identified quality criteria and aspects were extracted and clustered into quality dimensions.

Results

A total of 70 publications met inclusion criteria. Included publications contain information on five quality assurance systems and further 24 quality assessment tools for mHealth apps. Of these 29 systems/tools, 8 were developed for the assessment of mHealth apps for specific diseases, 16 for assessing mHealth apps for all fields of health and another five are not restricted to health apps. Identified quality criteria and aspects were extracted and grouped into a total of 14 quality dimensions, namely "information and transparency", "validity and (added) value", "(medical) safety", "interoperability and compatibility", "actuality", "engagement", "data privacy and data security", "usability and design", "technology", "organizational aspects", "social aspects", "legal aspects", "equity and equality", and "cost(-effectiveness)".

Discussion

This scoping review provides a broad overview of existing quality assessment and assurance systems. Many of the tools included cover only a few dimensions and aspects and therefore do not allow for a comprehensive quality assessment or quality assurance. Our findings can contribute to the development of continuous quality assessment and assurance systems for mHealth apps.

Systematic Review Registration

https://www.researchprotocols.org/2022/7/e36974/, International Registered Report Identifier, IRRID (DERR1-10.2196/36974).

Usability, Appeal, and Relevance of Music and Wellbeing Training for Carers of People with Dementia: A Think-Aloud Study

Background

Personalized music listening can be highly effective in supporting people experiencing changes in mood and behavior due to Alzheimer's disease and other conditions causing dementia. However, a lack of staff education can be a barrier to effective use of music in residential aged care.

Objective

This study investigates the usability, appeal and relevance of an online training course designed to help care staff develop personalized music programs for people living with dementia in their care.

Methods

A mixed methods approach was taken in which 13 participants took part in a think-aloud session while using the training. Qualitative data from the think-aloud session and interviews were triangulated with results of a survey evaluation of the training.

Results

Themes relating to Engagement, Usability and Appeal, Pedagogical Design and Content, and Transfer, Impact and Barriers to Uptake were discussed. Results indicated that participants regarded the virtual environment and pedagogical design as of high quality, although some minor issues in navigability were identified. Participants also found the training to be highly relevant to their roles as caregivers and reported ways they would incorporate key concepts into care practices. However, ongoing barriers to implementation of such training were identified including systemic issues that contribute to a culture of focus on physical care.

Conclusions

Consistency of navigation and clear instructions for interacting with content are important in training of time-poor care staff. Linking training to industry standards can provide important motivation for implementation of training, although systemic barriers can impede real-world change.

Benchmarking usability of patient portals in Estonia, Finland, Norway, and Sweden

INTRODUCTION

Poor usability is a barrier to widespread adoption of electronic health records (EHR). Providing good usability is especially challenging in the health care context, as there is a wide variety of patient users. Usability benchmarking is an approach for improving usability by evaluating and comparing the strength and weaknesses of systems. The main purpose of this study is to benchmark usability of patient portals across countries.

METHODS

A mixed-methods survey approach was applied to benchmark the national patient portals offering patient access to EHR in Estonia, Finland, Norway, and Sweden. These Nordic countries have similar public healthcare systems, and they are pioneers in offering patients access to EHR for several years. In a survey of 29,334 patients, both patients' quantitative ratings of usability and their qualitative descriptions of very positive and very negative peak experiences of portal use were collected.

RESULTS

The usability scores ranged from good to fair level of usability. The narratives of very positive and very negative experiences included the benefits of the patient portals and experienced usability issues. The regression analysis of results showed that very positive and negative experiences of patient portal use explain 19-35% of the variation of usability scores in the four countries. The percentage of patients who reported very positive or very negative experiences in each country was unrelated to the usability scores across countries.

CONCLUSIONS

The survey approach could be used to evaluate usability with a wide variety of users and it supported learning from comparison across the countries. The combination of quantitative and qualitative data provided an approximation of the level of the perceived usability, and identified usability issues to be improved and useful features that patients appreciate. Further work is needed to improve the comparability of the varied samples across countries.

A web-based resource for exercise training in children treated for brain tumours to improve cognitive sequelae: Development and usability

Improving cognitive sequelae in children treated for brain tumours (CTBT) requires accessible interventions. While instructor-led exercise in a hospital setting is efficacious, it is not extended to communities.

Objectives

We aimed to (i) develop a website with educational resources/tools for community health and fitness professionals (HFP) to deliver exercise for CTBT in community settings to improve cognition and (ii) assess its usability by community HFP. It was hypothesized that the website would be learnable, clear, satisfactory and efficient to deliver exercise.

Methods

A scoping review determined the state of eHealth resources to support exercise for CTBT and identified knowledge and resource gaps. Three focus groups with HFP who served cancer survivors in hospital or community settings (n = 13) identified user needs; content analysis identified themes. Gaps from the scoping review and themes from focus groups informed website content. A questionnaire assessed its usability by community HFP (n = 4). Descriptive statistics inferred the website's learnability, clarity, satisfaction and efficiency. Open-ended responses identified issues.

Results

The scoping review revealed a lack of eHealth resources supporting exercise to improve cognition in CTBT and education for HFP to deliver exercise. Six themes were identified in the focus groups. HFP rated the website as sufficiently learnable, clear, satisfactory and efficient. Two minor issues were reported and addressed.

Conclusion

The website marks one of the first eHealth resources to increase accessibility of intervention to improve cognitive sequelae and ultimately quality of life in CTBT. HFP also gain access to education and tools to deliver exercise in community settings.

Patient and Clinician Perceptions of the Pulse Oximeter in a Remote Monitoring Setting for COVID-19: Qualitative Study

BACKGROUND

As a response to the COVID-19 pandemic, the Sydney Local Health District in New South Wales, Australia, launched the rpavirtual program, the first full-scale virtual hospital in Australia, to remotely monitor and follow up stable patients with COVID-19. As part of the intervention, a pulse oximeter wearable device was delivered to patients to monitor their oxygen saturation levels, a critical indicator of COVID-19 patient deterioration. Understanding users' perceptions toward the device is fundamental to assessing its usability and acceptability and contributing to the effectiveness of the intervention, but no research to date has explored the user experience of the pulse oximeter for remote monitoring in this setting.

OBJECTIVE

This study aimed to explore the use, performance, and acceptability of the pulse oximeter by clinicians and patients in rpavirtual during COVID-19.

METHODS

Semistructured interviews and usability testing were conducted. Stable adult patients with COVID-19 (aged ≥18 years) who used the pulse oximeter and were monitored by rpavirtual, and rpavirtual clinicians monitoring these patients were interviewed. Clinicians could be nurses, doctors, or staff who were part of the team that assisted patients with the use of the pulse oximeter. Usability testing was conducted with patients who had the pulse oximeter when they were contacted. Interviews were coded using the Theoretical Framework of Acceptability. Usability testing was conducted using a think-aloud protocol. Data were collected until saturation was reached.

RESULTS

Twenty-one patients (average age 51, SD 13 years) and 15 clinicians (average age 41, SD 11 years) completed the interview. Eight patients (average age 51, SD 13 years) completed the usability testing. All participants liked the device and thought it was easy to use. They also had a good understanding of how to use the device and the device's purpose. Patients' age and device use-related characteristics (eg, the warmth of hands and hand steadiness) were identified by users as factors negatively impacting the accurate use of the pulse oximeter.

CONCLUSIONS

Patients and clinicians had very positive perceptions of the pulse oximeter for COVID-19 remote monitoring, indicating high acceptability and usability of the device. However, factors that may impact the accuracy of the device should be considered when delivering interventions using the pulse oximeter for remote monitoring. Targeted instructions about the use of the device may be necessary for specific populations (eg, older people and patients unfamiliar with technology). Further research should focus on the integration of the pulse oximeter data into electronic medical records for real-time and secure patient monitoring.

A Tablet-Based App to Support Nursing Home Staff in Delivering an Individualized Cognitive and Physical Exercise Program for Individuals With Dementia: Mixed Methods Usability Study

BACKGROUND

The promotion of physical activity in individuals with dementia living in nursing homes is crucial for preserving physical and cognitive functions and the associated quality of life. Nevertheless, the implementation of physical activity programs in this setting is challenging, as the time and expertise of nursing home staff are limited. This situation was further exacerbated by the COVID-19 pandemic. Mobile health apps may be a sustainable approach to overcome these challenges in the long term. Therefore, the Individualized Cognitive and Physical Exercise-App (the InCoPE-App) was developed to support nursing home staff in delivering and implementing tailored cognitive and physical exercise training for individuals with dementia.

OBJECTIVE

This study aims to assess the usability of the InCoPE-App in terms of user performance and user perception in a laboratory setting using a mixed methods approach.

METHODS

Nursing home staff were encouraged to perform 5 basic tasks within the InCoPE-App. Their thoughts while using the app were captured by implementing a think aloud protocol. Then, participants completed the System Usability Scale questionnaire. The think aloud transcripts were qualitatively evaluated to unveil usability issues. All identified issues were rated in terms of their necessity to be fixed. Task completion (ie, success rate and time) and perceived usability were evaluated descriptively.

RESULTS

A total of 14 nursing home employees (mean age 53.7, SD 10.6 years; n=13, 93% women) participated in the study. The perceived usability of the InCoPE-App, as assessed by the System Usability Scale questionnaire, can be rated as "good." The main usability issues concerned navigation logic and comprehensibility of app content.

CONCLUSIONS

The InCoPE-App is a user-friendly app that enables nursing home staff to deliver and implement cognitive and physical exercise training for individuals with dementia in nursing homes. The InCoPE-App can be used with little training, even by people aged ≥50 years, who may have low digital literacy. To achieve sustainable use and high user satisfaction of the InCoPE-App in the long term, it should be implemented and evaluated in a field study.

Perceived Use Cases, Barriers, and Requirements for a Smart Health-Tracking Toilet Seat: Qualitative Focus Group Study

BACKGROUND

Smart bathroom technology offers unrivaled opportunities for the automated measurement of a range of biomarkers and other data. Unfortunately, efforts in this area are mostly driven by a technology push rather than market pull approach, which decreases the chances of successful adoption. As yet, little is known about the use cases, barriers, and desires that potential users of smart bathrooms perceive.

OBJECTIVE

This study aimed to investigate how participants from the general population experience using a smart sensor-equipped toilet seat installed in their home. The study contributes to answering the following questions: What use cases do citizens see for this innovation? and What are the limitations and barriers to its everyday use that they see, including concerns regarding privacy, the lack of fit with everyday practices, and unmet expectations for user experience?

METHODS

Overall, 31 participants from 30 households participated in a study consisting of 3 (partially overlapping) stages: sensitizing, in which participants filled out questionnaires to trigger their thoughts about smart bathroom use and personal health; provotyping, in which participants received a gentle provocation in the form of a smart toilet seat, which they used for 2 weeks; and discussion, in which participants took part in a web-based focus group session to discuss their experiences.

RESULTS

Participants mostly found the everyday use of the toilet, including installation and dismantling when necessary, to be relatively easy and free of complications. Where complications occurred, participants mentioned issues related to the design of the prototype, technology, or mismatches with normal practices in using toilets and hygiene. A broad range of use cases were mentioned, ranging from signaling potentially detrimental health conditions or exacerbations of existing conditions to documenting physical data to measuring biomarkers to inform a diagnosis and behavioral change. Participants differed greatly in whether they let others use, or even know about, the seat. Ownership and control over their own data were essential for most participants.

CONCLUSIONS

This study showed that participants felt that a smart toilet seat could be acceptable and effective, as long as it fits everyday practices concerning toilet use and hygiene. The range of potential uses for a smart toilet seat is broad, as long as privacy and control over disclosure and data are warranted.

A usability evaluation of the perceived user friendliness, accessibility, and inclusiveness of a personalized digital care pathway tool

OBJECTIVE

This study aimed to acquire insight into the perceived user friendliness, accessibility and inclusiveness of a personalized digital care pathway.

MATERIALS & METHODS

Usability of the tool was tested in an experimental setting. Mixed methods data collection consisted of scenario-based eye tracking tests in a web- or mobile-based prototype of the tool, followed by a questionnaire assessing user friendliness (System Usability Scale; SUS) and a structured interview. Inclusiveness was assessed by subgroup comparisons based on language proficiency, age and education level. Via purposive sampling a heterogeneous population of users (N = 24) was recruited. Eye tracking was used to measure gaze behavior.

RESULTS

Overall, participants were satisfied with the tool (scale 0-10, 7.5; SD = 1.29). User friendliness of the mobile version (68.3; SD = 21.6) was higher than the web version (50.9; SD = 17.3) measured by SUS score (0-100). With regard to accessibility, eye tracking scenarios showed that the menu bar was hard to find (17% mobile, 55% web). In all scenario's, information was found faster in the mobile version than the web version. Attention was easily drawn to images. Regarding inclusiveness of the tool, we found significantly longer completing time of the scenario tasks for low language proficiency (p-value = 0.029) and higher age subgroups (p-value = 0.049). Lower language proficiency scored a significant lower SUS score (p-value = 0.012).

CONCLUSIONS

Overall, user friendliness and accessibility were positively evaluated. Assessment of inclusiveness emphasized the need for tailoring digital tools to those with low language proficiency and/or an older age. Co-creation of digital care tools with users is therefore important to match users' needs, make tools easily understandable and accessible to all users, and ultimately result in better uptake and impact.

Developing and Testing a Smartphone Application to Enhance Adherence to Voice Therapy: A Pilot Study

The present study aimed to develop a smartphone application (app) that addressed identified barriers to success in voice therapy; accessibility, and poor adherence to home practice. The study objectives were (1) to investigate if app use enhanced adherence to the home practice of voice therapy and (2) to test app usability. Maximizing the effectiveness of voice therapy is vital as voice disorders are detrimental to personal and professional quality of life. A single-blinded randomized clinical trial was completed for the first objective. Participants included normophonic individuals randomly assigned to the app group or the traditional group. The primary outcome measure was adherence measured as the number of missed home practice tasks. The second objective was completed through usability testing and a focus group discussion. The app group (n = 12) missed approximately 50% less home practice tasks as compared to the traditional group (n = 13) and these results were statistically significant (p = 0.04). Dropout rates were comparable between the two groups. Usability results were positive for good usability with high perceived usefulness and perceived ease of use. App use resulted in improved adherence to home practice tasks. App usability results were positive, and participants provided specific areas of improvement which are achievable. Areas for improvement include app engagement and willingness to pay.

Design and Evaluation of Personalized Services to Foster Active Aging: The Experience of Technology Pre-Validation in Italian Pilots

Assistive devices could promote independent living and support the active and healthy aging of an older population; however, several factors can badly influence the long-term use of new technologies. In this context, this paper presents a two-step methodology called “pre-validation” that aims to identify the factors that can bias the use of new services, thus minimizing the risk of an unsuccessful longer trial. The proposed pre-validation methodology is composed of two main phases that aim to assess the usability and the reliability of the technology assessed in a laboratory environment and the usability, acceptability, user experience, and reliability of the technology in real environments. The tested services include the socialization scenario, in which older adults are better connected to the community via technological solutions (i.e., socialization applications), and the monitoring scenario, which allows for the introduction of timely interventions (technologies involved include environmental monitoring sensors, a telepresence robot, wearable sensors, and a personalized dashboard). The obtained results underline an acceptable usability level (average System Usability Scale score > 65) for the tested technologies (i.e., socialization applications and a telepresence robot). Phase Two also underlines the good acceptability, user experience, and usability of the tested services. The statistical analysis underlines a correlation between the stress related to the use of technology, digital skills, and intention of use, among other factors. Qualitative feedback also remarks on a correlation between older adults with low digital skills and an anxiety about using technology. Positive correlation indexes were highlighted between the trust and usability scores. Eventually, future long-term trials with assistive technology should rely on motivated caregivers, be founded on a strong recruitment process, and should reassure older adults—especially the ones with low digital literacy—about the use of technology by proposing personalized training and mentoring, if necessary, to increase the trust.

Assessing the Acceptability of Home Blood Monitoring for Patients With Cancer Who Are Receiving Systemic Anticancer Therapy From a Patient, Caregiver, and Clinician Perspective: Focus Group and Interview Study

BACKGROUND

Regular blood testing is an integral part of systemic anticancer therapy delivery. Blood tests are required before every administration of treatment to ensure that a patient is sufficiently well to receive it. Blood testing is burdensome for patients as they require either an extra visit within 48 hours of planned administration of treatment or a significantly long visit if performed on the day of treatment. The additional time for appointments can have a significant impact on the quality of life of someone who is living with cancer. In the United Kingdom, the COVID-19 pandemic created unprecedented disruption to the delivery of cancer care. Face-to-face hospital visits were reduced, resulting in the need to develop more innovative ways of working to minimize treatment interruptions. This led to significant uptake of digital technologies, with new models of care rapidly deployed across the UK health service to meet these challenges.

OBJECTIVE

This study aimed to explore the acceptability of a point-of-care home blood monitoring device for people with cancer who are receiving systemic anticancer therapy, which is being developed in response to the increased need for remote care for patients with cancer.

METHODS

Qualitative focus groups and semistructured interviews were conducted with patients (23/47, 49%), caregivers (6/47, 13%), and health care professionals (18/47, 38%) over a 19-month time frame from May 2019 to December 2020. Data were analyzed using framework analysis guided by the Unified Theory of Acceptance and Use of Technology model.

RESULTS

Analysis identified 4 overarching themes: performance expectancy, effort expectancy, social influence, and facilitating conditions.

CONCLUSIONS

This study found that patients with cancer, their caregivers, and health care professionals had positive perceptions about home blood monitoring. Although they are often considered synonymously, self-testing and self-management are not mutually exclusive, and this study illustrated some disparity in opinions regarding patient self-management. Home blood monitoring has the potential to provide patients with cancer with a convenient option for blood monitoring. It would minimize hospital attendances, decrease late treatment deferrals, and provide prompt recognition of cancer treatment toxicities, thus enhancing the existing nurse-led protocols and clinical pathways. Home blood monitoring would create a long-term sustainable transformation for the delivery of cancer care, using digital health to act as a facilitator to address a pertinent issue regarding improving the efficiency of hospital resources and increasing the delivery of personalized patient care. Further studies are needed to determine how and where home blood monitoring would fit within clinical pathways, in a way that is robust and equitable.

Web version of the protocol of the orofacial myofunctional evaluation with scores: usability and learning

PURPOSE

The Orofacial Myofunctional Evaluation with Scores (OMES) protocol has been validated and used in clinical practice and research. The goals of this study were to develop, analyze and improve a version of OMES for the Web and to investigate the relationship between the usability judgments and the prior experience of the evaluators and whether using the interface promotes learning, as shown by the task completion time (TCT).

METHODS

Study steps: 1) inspection of the prototype by the team; 2) evaluation of usability by three experienced speech-language pathologists (SLPs); and 3) evaluation of its usability by 12 SLPs with varying levels of experience in the use of OMES. Participants answered the Heuristic evaluation (HE), the Computer System Usability Questionnaire (CSUQ), and expressed free comments. The TCT was recorded.

RESULTS

The OMES-Web reached excellent usability levels, and the participants were highly satisfied. The correlations between the participants' experience and the HE and CSUQ scores were not significant. The TCT decreased significantly throughout the tasks.

CONCLUSION

OMES-Web meets the usability criteria, and participants feel satisfied with the system regardless of their level of experience. The fact that it is easy to learn favors its adoption by professionals.

Usability evaluation of mHealth apps for elderly individuals: a scoping review

BACKGROUND

Usability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.

METHODS

Electronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.

RESULTS

Ninety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.

CONCLUSION

In the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Measuring the Usability of eHealth Solutions for Patients With Parkinson Disease: Observational Study

Background

Parkinson disease (PD) is a neurodegenerative disorder with a variety of motor and nonmotor symptoms. Many of these symptoms can be monitored by eHealth solutions, including smartphone apps, wearable sensors, and camera systems. The usability of such systems is a key factor in long-term use, but not much is known about the predictors of successful use and preferable methods to assess usability in patients with PD.

Objective

This study tested methods to assess usability and determined prerequisites for successful use in patients with PD.

Methods

We performed comprehensive usability assessments with 18 patients with PD using a mixed methods usability battery containing the System Usability Scale, a rater-based evaluation of device-specific tasks, and qualitative interviews. Each patient performed the usability battery with 2 of 3 randomly assigned devices: a tablet app, wearable sensors, and a camera system. The usability battery was administered at the beginning and at the end of a 4-day testing period. Between usability batteries, the systems were used by the patients during 3 sessions of motor assessments (wearable sensors and camera system) and at the movement disorder ward (tablet app).

Results

In this study, the rater-based evaluation of tasks discriminated the best between the 3 eHealth solutions, whereas subjective modalities such as the System Usability Scale were not able to distinguish between the systems. Successful use was associated with different clinical characteristics for each system: eHealth literacy and cognitive function predicted successful use of the tablet app, and better motor function and lower age correlated with the independent use of the camera system. The successful use of the wearable sensors was independent of clinical characteristics. Unfortunately, patients who were not able to use the devices well provided few improvement suggestions in qualitative interviews.

Conclusions

eHealth solutions should be developed with a specific set of patients in mind and subsequently tested in this cohort. For a complete picture, usability assessments should include a rater-based evaluation of task performance, and there is a need to develop strategies to circumvent the underrepresentation of poorly performing patients in qualitative usability research.

Reliability and Usability Analysis of an Embedded System Capable of Evaluating Balance in Elderly Populations Based on a Modified Wii Balance Board

This paper analyzes the reliability and usability of a portable electronic instrument that measures balance and balance impairment in older adults. The center of pressure (CoP) metrics are measured with a modified Wii Balance Board (mWBB) platform. In the intra- and inter-rater testing, 16 and 43 volunteers (mean 75.66 and standard deviation (SD) of 7.86 years and 72.61 (SD 7.86) years, respectively) collaborated. Five volunteer raters (5.1 (SD 3.69) years of experience) answered the System Usability Scale (SUS). The most reliable CoP index in the intra-examiner tests was the 95% power frequency in the medial-lateral displacement of the CoP with closed-eyes. It had excellent reliability with an intraclass correlation coefficient ICC = 0.948 (C.I. 0.862-0.982) and a Pearson's correlation coefficient PCC = 0.966 (p < 0.001). The best index for the inter-rater reliability was the centroidal frequency in the anterior-posterior direction closed-eyes, which had an ICC (2,1) = 0.825. The mWBB also obtained a high usability score. These results support the mWBB as a reliable complementary tool for measuring balance in older adults. Additionally, it does not have the limitations of laboratory-grade systems and clinical screening instruments.

Usability-Focused Development and Usage of NeoTree-Beta, an App for Newborn Care in a Low-Resource Neonatal Unit, Malawi

Background

Neonatal mortality is high in low-resource settings. NeoTree is a digital intervention for neonatal healthcare professionals (HCPs) aiming to achieve data-driven quality improvement and improved neonatal survival in low-resource hospitals. Optimising usability with end-users could help digital health interventions succeed beyond pilot stages in low-resource settings. Usability is the quality of a user's experience when interacting with an intervention, encompassing their effectiveness, efficiency, and overall satisfaction.

Objective

To evaluate the usability and usage of NeoTree beta-app and conduct Agile usability-focused intervention development.

Method

A real-world pilot of NeoTree beta-app was conducted over 6 months at Kamuzu Central Hospital neonatal unit, Malawi. Prior to deployment, think-aloud interviews were conducted to guide nurses through the app whilst voicing their thoughts aloud (n = 6). System Usability Scale (SUS) scores were collected before the implementation of NeoTree into usual clinical care and 6 months after implementation (n = 8 and 8). During the pilot, real-world user-feedback and user-data were gathered. Feedback notes were subjected to thematic analysis within an Agile “product backlog.” For usage, number of users, user-cadre, proportion of admissions/outcomes recorded digitally, and median app-completion times were calculated.

Results

Twelve overarching usability themes generated 57 app adjustments, 39 (68%) from think aloud analysis and 18 (32%) from the real-world testing. A total of 21 usability themes/issues with corresponding app features were produced and added to the app. Six themes relating to data collection included exhaustiveness of data schema, prevention of errors, ease of progression, efficiency of data entry using shortcuts, navigation of user interface (UI), and relevancy of content. Six themes relating to the clinical care included cohesion with ward process, embedded education, locally coherent language, adaptability of user-interface to available resources, and printout design to facilitate handover. SUS scores were above average (88.1 and 89.4 at 1 and 6 months, respectively). Ninety-three different HCPs of 5 cadres, recorded 1,323 admissions and 1,197 outcomes over 6 months. NeoTree achieved 100% digital coverage of sick neonates admitted. Median completion times were 16 and 8 min for admissions and outcomes, respectively.

Conclusions

This study demonstrates optimisation of a digital health app in a low-resource setting and could inform other similar usability studies apps in similar settings.

A Veteran-Centric Web-Based Decision Aid for Lung Cancer Screening: Usability Analysis

BACKGROUND

Web-based tools developed to facilitate a shared decision-making (SDM) process may facilitate the implementation of lung cancer screening (LCS), an evidence-based intervention to improve cancer outcomes. Veterans have specific risk factors and shared experiences that affect the benefits and potential harms of LCS and thus may value a veteran-centric LCS decision tool (LCSDecTool).

OBJECTIVE

This study aims to conduct usability testing of an LCSDecTool designed for veterans receiving care at a Veteran Affairs medical center.

METHODS

Usability testing of the LCSDecTool was conducted in a prototype version (phase 1) and a high-fidelity version (phase 2). A total of 18 veterans and 8 clinicians participated in phase 1, and 43 veterans participated in phase 2. Quantitative outcomes from the users included the System Usability Scale (SUS) and the End User Computing Satisfaction (EUCS) in phase 1 and the SUS, EUCS, and Patient Engagement scale in phase 2. Qualitative data were obtained from observations of user sessions and brief interviews. The results of phase 1 informed the modifications of the prototype for the high-fidelity version. Phase 2 usability testing took place in the context of a pilot hybrid type 1 effectiveness-implementation trial.

RESULTS

In the phase 1 prototype usability testing, the mean SUS score (potential range: 0-100) was 81.90 (SD 9.80), corresponding to an excellent level of usability. The mean EUCS score (potential range: 1-5) was 4.30 (SD 0.71). In the phase 2 high-fidelity usability testing, the mean SUS score was 65.76 (SD 15.23), corresponding to a good level of usability. The mean EUCS score was 3.91 (SD 0.95); and the mean Patient Engagement scale score (potential range 1 [low] to 5 [high]) was 4.62 (SD 0.67). The median time to completion in minutes was 13 (IQR 10-16). A thematic analysis of user statements documented during phase 2 high-fidelity usability testing identified the following themes: a low baseline level of awareness and knowledge about LCS increased after use of the LCSDecTool; users sought more detailed descriptions about the LCS process; the LCSDecTool was generally easy to use, but specific navigation challenges remained; some users noted difficulty understanding medical terms used in the LCSDecTool; and use of the tool evoked veterans' struggles with prior attempts at smoking cessation.

CONCLUSIONS

Our findings support the development and use of this eHealth technology in the primary care clinical setting as a way to engage veterans, inform them about a new cancer control screening test, and prepare them to participate in an SDM discussion with their provider.

The Role of Health Kiosks: Scoping Review

BACKGROUND

Health kiosks are publicly accessible computing devices that provide access to services, including health information provision, clinical measurement collection, patient self-check-in, telemonitoring, and teleconsultation. Although the increase in internet access and ownership of smart personal devices could make kiosks redundant, recent reports have predicted that the market will continue to grow.

OBJECTIVE

We seek to clarify the current and future roles of health kiosks by investigating the settings, roles, and clinical domains in which kiosks are used; whether usability evaluations of health kiosks are being reported, and if so, what methods are being used; and what the barriers and facilitators are for the deployment of kiosks.

METHODS

We conducted a scoping review using a bibliographic search of Google Scholar, PubMed, and Web of Science databases for studies and other publications between January 2009 and June 2020. Eligible papers described the implementation as primary studies, systematic reviews, or news and feature articles. Additional reports were obtained by manual searching and querying the key informants. For each article, we abstracted settings, purposes, health domains, whether the kiosk was opportunistic or integrated with a clinical pathway, and whether the kiosk included usability testing. We then summarized the data in frequency tables.

RESULTS

A total of 141 articles were included, of which 134 (95%) were primary studies, and 7 (5%) were reviews. Approximately 47% (63/134) of the primary studies described kiosks in secondary care settings. Other settings included community (32/134, 23.9%), primary care (24/134, 17.9%), and pharmacies (8/134, 6%). The most common roles of the health kiosks were providing health information (47/134, 35.1%), taking clinical measurements (28/134, 20.9%), screening (17/134, 12.7%), telehealth (11/134, 8.2%), and patient registration (8/134, 6.0%). The 5 most frequent health domains were multiple conditions (33/134, 24.6%), HIV (10/134, 7.5%), hypertension (10/134, 7.5%), pediatric injuries (7/134, 5.2%), health and well-being (6/134, 4.5%), and drug monitoring (6/134, 4.5%). Kiosks were integrated into the clinical pathway in 70.1% (94/134) of studies, opportunistic kiosks accounted for 23.9% (32/134) of studies, and in 6% (8/134) of studies, kiosks were used in both. Usability evaluations of kiosks were reported in 20.1% (27/134) of papers. Barriers (e.g., use of expensive proprietary software) and enablers (e.g., handling of on-demand consultations) of deploying health kiosks were identified.

CONCLUSIONS

Health kiosks still play a vital role in the health care system, including collecting clinical measurements and providing access to web-based health services and information to those with little or no digital literacy skills and others without personal internet access. We identified research gaps, such as training needs for teleconsultations and scant reporting on usability evaluation methods.

A Remote Patient-Monitoring System for Intensive Care Medicine: Mixed Methods Human-Centered Design and Usability Evaluation

Background

Continuous monitoring of vital signs is critical for ensuring patient safety in intensive care units (ICUs) and is becoming increasingly relevant in general wards. The effectiveness of health information technologies such as patient-monitoring systems is highly determined by usability, the lack of which can ultimately compromise patient safety. Usability problems can be identified and prevented by involving users (ie, clinicians).

Objective

In this study, we aim to apply a human-centered design approach to evaluate the usability of a remote patient-monitoring system user interface (UI) in the ICU context and conceptualize and evaluate design changes.

Methods

Following institutional review board approval (EA1/031/18), a formative evaluation of the monitoring UI was performed. Simulated use tests with think-aloud protocols were conducted with ICU staff (n=5), and the resulting qualitative data were analyzed using a deductive analytic approach. On the basis of the identified usability problems, we conceptualized informed design changes and applied them to develop an improved prototype of the monitoring UI. Comparing the UIs, we evaluated perceived usability using the System Usability Scale, performance efficiency with the normative path deviation, and effectiveness by measuring the task completion rate (n=5). Measures were tested for statistical significance using a 2-sample t test, Poisson regression with a generalized linear mixed-effects model, and the N-1 chi-square test. P<.05 were considered significant.

Results

We found 37 individual usability problems specific to monitoring UI, which could be assigned to six subcodes: usefulness of the system, response time, responsiveness, meaning of labels, function of UI elements, and navigation. Among user ideas and requirements for the UI were high usability, customizability, and the provision of audible alarm notifications. Changes in graphics and design were proposed to allow for better navigation, information retrieval, and spatial orientation. The UI was revised by creating a prototype with a more responsive design and changes regarding labeling and UI elements. Statistical analysis showed that perceived usability improved significantly (System Usability Scale design A: mean 68.5, SD 11.26, n=5; design B: mean 89, SD 4.87, n=5; P=.003), as did performance efficiency (normative path deviation design A: mean 8.8, SD 5.26, n=5; design B: mean 3.2, SD 3.03, n=5; P=.001), and effectiveness (design A: 18 trials, failed 7, 39% times, passed 11, 61% times; design B: 20 trials, failed 0 times, passed 20 times; P=.002).

Conclusions

Usability testing with think-aloud protocols led to a patient-monitoring UI with significantly improved usability, performance, and effectiveness. In the ICU work environment, difficult-to-use technology may result in detrimental outcomes for staff and patients. Technical devices should be designed to support efficient and effective work processes. Our results suggest that this can be achieved by applying basic human-centered design methods and principles.

Trial Registration

ClinicalTrials.gov NCT03514173; https://clinicaltrials.gov/ct2/show/NCT03514173

Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation

Background  Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods.

Objective  Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations.

Methods  The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness.

Results  Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations.

Conclusion  We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.

Improving usability benchmarking for the eHealth domain: The development of the eHealth UsaBility Benchmarking instrument (HUBBI)

Background

Currently, most usability benchmarking tools used within the eHealth domain are based on re-classifications of old usability frameworks or generic usability surveys. This makes them outdated and not well suited for the eHealth domain. Recently, a new ontology of usability factors was developed for the eHealth domain. It consists of eight categories: Basic System Performance (BSP), Task-Technology Fit (TTF), Accessibility (ACC), Interface Design (ID), Navigation & Structure (NS), Information & Terminology (IT), Guidance & Support (GS) and Satisfaction (SAT).

Objective

The goal of this study is to develop a new usability benchmarking tool for eHealth, the eHealth UsaBility Benchmarking Instrument (HUBBI), that is based on a new ontology of usability factors for eHealth.

Methods

First, a large item pool was generated containing 66 items. Then, an online usability test was conducted, using the case study of a Dutch website for general health advice. Participants had to perform three tasks on the website, after which they completed the HUBBI. Using Partial Least Squares Structural Equation Modelling (PLS-SEM), we identified the items that assess each factor best and that, together, make up the HUBBI.

Results

A total of 148 persons participated. Our selection of items resulted in a shortened version of the HUBBI, containing 18 items. The category Accessibility is not included in the final version, due to the wide range of eHealth services and their heterogeneous populations. This creates a constantly different role of Accessibility, which is a problem for a uniform benchmarking tool.,

Conclusions

The HUBBI is a new and comprehensive usability benchmarking tool for the eHealth domain. It assesses usability on seven domains (BSP, TTF, ID, NS, IT, GS, SAT) in which a score per domain is generated. This can help eHealth developers to quickly determine which areas of the eHealth system’s usability need to be optimized.

Evaluation of a virtual coaching system eHealth intervention: A mixed methods observational cohort study in the Netherlands

BACKGROUND

With the rise in human life expectancy, the prevalence of chronic disease has increased significantly. Adopting a healthy lifestyle can decrease the risk of chronic disease. Virtual coaching systems can help older adults adopt a healthy lifestyle.AimThe primary objective of this study was to assess the use, user experience and potential health effects of a conversational agent-based eHealth platform (Council of Coaches) implemented in a real-world setting among older adults.

METHODS

An observational cohort study was conducted with older adults aged 55 years or older in the Netherlands. Participants were enrolled for 5-9 weeks during which they had access to Council of Coaches. They completed three questionnaires: pre-test, post-test, and at follow-up. After five weeks, an interview was conducted, and participants chose whether they wanted to use the eHealth intervention for another four weeks during the facultative phase.

RESULTS

The study population consisted of 51 older adults (70.6% female) with a mean age of 65.3 years (SD = 7.4). Of these, 94.1% started interacting with Council of Coaches, and most participants interacted once per week. During the facultative phase, 21 participants were still interacting with Council of Coaches. Minimal clinical important differences in quality of life were found among the study population after interacting with Council of Coaches.

CONCLUSION

Our results demonstrate that eHealth interventions with virtual coaching can be used among older adults. This may increase quality of life for older adults, and decrease their healthcare needs. Future research into such eHealth interventions should take into account the inclusion of sufficient personalised content and the use of a mixed methods study for assessing the eHealth intervention.

A Mixed-Method Approach to Develop and Validate an Integrated Food Literacy Tool for Personalized Food Literacy Guidance

Background

Food literacy refers to all practicalities associated with healthy eating. Current food literacy tools are limited in practical use in clinical practice. Therefore, an integrated food literacy tool (IFLT) to assess food literacy and to personalize food literacy guidance was developed and validated.

Methods

Following an iterative process, a food literacy framework was developed and food literacy goals were defined. A corresponding food literacy screener (FLS) to assess food literacy was developed along with an algorithm to provide personalized food literacy guidance based on the food literacy assessment. Content validation of the FLS was assessed by a panel of experts, measuring item and scale content validity index (I/S-CVI) and by the target population in semi structured interviews with 15 adults of reproductive age. Subsequently, an online cross-sectional survey was conducted among 114 adults of reproductive age to evaluate the validity of the FLS. Construct validity was examined against both the validated healthy eating and weight self-efficacy scale and against a food frequency questionnaire assessing healthy eating self-efficacy (HESE) and diet quality, respectively. Reliability was assessed with a two-week test-retest. Pearson correlation tests were conducted.

Results

An IFLT consisting of a FLS and corresponding algorithm to personalize food literacy guidance by prioritizing food literacy goals was developed. The IFLT includes 24 food literacy goals, addressed by 17 FLS items. Every item received a weighting factor based on theory and expert opinion to prioritize food literacy goals according to personal needs. Content validity revealed that the FLS was rated relevant by experts (S-CVI = 0.93) and well-understood by the target population. The FLS has a good construct validity as it was positively correlated with diet quality (r = 0.536, p &lt; 0.001) and with HESE (r = 0.685, p &lt; 0.001). It also showed a good test-retest reliability (r = 0.721, p &lt; 0.001).

Conclusion

The newly developed IFLT is a practically applicable, context specific theory-and expert-based dual purpose tool to assess food literacy and to personalize food literacy guidance by prioritizing individuals' food literacy goals to their needs.

Assessments of a novel digital follow-up tool Rehabkompassen® to identify rehabilitation needs among stroke patients in an outpatient setting

Introduction

It remains a huge challenge to identify individual rehabilitation needs in a time-efficient manner for providing patient-tailored rehabilitation during the continuum of stroke care. We have recently demonstrated the usefulness of a paper-version Rehab-Compass as a follow-up tool. The aim of the current study was to develop a digital version of the Rehab-Compass and evaluate its usability and feasibility.

Methods

The novel digital tool Rehabkompassen® was developed by an iterative and participatory design process. Patients' rehabilitation needs were visualized by the tool and used before, during, and after the consultation. The usability and feasibility of the tool was assessed by task completion rate, the System Usability Scale, and satisfaction questionnaires among 2 physicians and 24 adult stroke patients in an outpatient clinical setting.

Results

Rehabkompassen® identified and graphically visualized a panoramic view of the stroke patients' multidimensional needs in individual- and group levels. The instrument appeared to be feasible and time efficient in clinical use with a 100% overall task completion rate for both patients and physicians. A majority of the patients reported that it was very easy or fairly easy to answer the digital questionnaires and to understand their own digital Rehab-Compass graph. Two physicians reported a high mean score on the System Usability Scale (95/100) and were positive about using the tool in the future.

Conclusions

The current results indicated that Rehabkompassen® was a feasible, useful, and time-saving follow-up tool for the identification of rehabilitation needs among stroke survivors in the post-acute continuum of care after stroke. Further research is needed to evaluate the efficacy of the digital instrument among stroke patients.

Usability Evaluation of an Offline Electronic Data Capture App in a Prospective Multicenter Dementia Registry (digiDEM Bayern): Mixed Method Study

BACKGROUND

Digital registries have been shown to provide an efficient way of gaining a better understanding of the clinical complexity and long-term progression of diseases. The paperless method of electronic data capture (EDC) during a patient interview saves both time and resources. In the prospective multicenter project "Digital Dementia Registry Bavaria (digiDEM Bayern)," interviews are also performed on site in rural areas with unreliable internet connectivity. It must be ensured that EDC can still be performed in such a context and that there is no need to fall back on paper-based questionnaires. In addition to a web-based data collection solution, the EDC system REDCap (Research Electronic Data Capture) offers the option to collect data offline via an app and to synchronize it afterward.

OBJECTIVE

The aim of this study was to evaluate the usability of the REDCap app as an offline EDC option for a lay user group and to examine the necessary technology acceptance of using mobile devices for data collection. The feasibility of the app-based offline data collection in the digiDEM Bayern dementia registry project was then evaluated before going live.

METHODS

An exploratory mixed method design was employed in the form of an on-site usability test with the "Thinking Aloud" method combined with an online questionnaire including the System Usability Scale (SUS). The acceptance of mobile devices for data collection was surveyed based on five categories of the technology acceptance model.

RESULTS

Using the "Thinking Aloud" method, usability issues were identified and solutions were accordingly derived. Evaluation of the REDCap app resulted in a SUS score of 74, which represents "good" usability. After evaluating the technology acceptance questionnaire, it can be concluded that the lay user group is open to mobile devices as interview tools.

CONCLUSIONS

The usability evaluation results show that a lay user group generally agree that data collecting partners in the digiDEM project can handle the REDCap app well. The usability evaluation provided statements about positive aspects and could also identify usability issues relating to the REDCap app. In addition, the current technology acceptance in the sample showed that heterogeneous groups of different ages with diverse experiences in handling mobile devices are also ready for the use of app-based EDC systems. Based on these results, it can be assumed that the offline use of an app-based EDC system on mobile devices is a viable solution for collecting data in a decentralized registry-based research project.

Sociodemographic Characteristics Associated with an eHealth System Designed to Reduce Depressive Symptoms among Patients with Breast or Prostate Cancer: Prospective Study (Preprint)

Background

eHealth interventions have become a topic of interest in the field of mental health owing to their increased coordination and integration of different elements of care, in treating and preventing mental ill health in patients with somatic illnesses. However, poor usability, learnability, and user engagement might affect the effectiveness of an eHealth intervention. Identifying different sociodemographic characteristics that might be associated with higher perceived usability can help improve the usability of eHealth interventions.

Objective

This study aimed to identify the sociodemographic characteristics that might be associated with the perceived usability of the NEVERMIND (Neurobehavioural Predictive and Personalised Modelling of Depressive Symptoms During Primary Somatic Diseases) eHealth system, comprising a mobile app and a sensorized shirt, in reducing comorbid depressive symptoms in patients with breast or prostate cancer.

Methods

The study included a total of 129 patients diagnosed with breast (n=80, 62%) or prostate (n=49, 38%) cancer, who received a fully automated mobile app and sensorized shirt (NEVERMIND system). Sociodemographic data on age, sex, marital status, education level, and employment status were collected at baseline. Usability outcomes included the System Usability Scale (SUS), a subjective measure that covers different aspects of system usability; the user version of the Mobile App Rating Scale (uMARS), a user experience questionnaire; and a usage index, an indicator calculated from the number of days patients used the NEVERMIND system during the study period.

Results

The analysis was based on 108 patients (n=68, 63%, patients with breast cancer and n=40, 37%, patients with prostate cancer) who used the NEVERMIND system for an average of 12 weeks and completed the study. The overall mean SUS score at 12 weeks was 73.4 (SD 12.5), which indicates that the NEVERMIND system has good usability, with no statistical differences among different sociodemographic characteristics. The global uMARS score was 3.8 (SD 0.3), and women rated the app higher than men (β=.16; P=.03, 95% CI 0.02-0.3), after adjusting for other covariates. No other sociodemographic characteristics were associated with higher uMARS scores. There was a statistical difference in the use of the NEVERMIND system between women and men. Women had significantly lower use (β=–0.13; P=.04, 95% CI −0.25 to −0.01), after adjusting for other covariates.

Conclusions

The findings suggest that the NEVERMIND system has good usability according to the SUS and uMARS scores. There was a higher favorability of mobile apps among women than among men. However, men had significantly higher use of the NEVERMIND system. Despite the small sample size and low variability, there is an indication that the NEVERMIND system does not suffer from the digital divide, where certain sociodemographic characteristics are more associated with higher usability.

Trial Registration

German Clinical Trials Register RKS00013391; https://www.drks.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00013391

Use of Patient-reported Outcome Measures and Patient-reported Experience Measures within Evaluation Studies of Telemedicine Applications: Systematic Review

Background

With the rise of digital health technologies and telemedicine, the need for evidence-based evaluation is growing. Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) are recommended as an essential part of the evaluation of telemedicine. For the first time, a systematic review has been conducted to investigate the use of PROMs and PREMs in the evaluation studies of telemedicine covering all application types and medical purposes.

Objective

This study investigates the following research questions: in which scenarios are PROMs and PREMs collected for evaluation purposes, which PROM and PREM outcome domains have been covered and how often, which outcome measurement instruments have been used and how often, does the selection and quantity of PROMs and PREMs differ between study types and application types, and has the use of PROMs and PREMs changed over time.

Methods

We conducted a systematic literature search of the MEDLINE and Embase databases and included studies published from inception until April 2, 2020. We included studies evaluating telemedicine with patients as the main users; these studies reported PROMs and PREMs within randomized controlled trials, controlled trials, noncontrolled trials, and feasibility trials in English and German.

Results

Of the identified 2671 studies, 303 (11.34%) were included; of the 303 studies, 67 (22.1%) were feasibility studies, 70 (23.1%) were noncontrolled trials, 20 (6.6%) were controlled trials, and 146 (48.2%) were randomized controlled trials. Health-related quality of life (n=310; mean 1.02, SD 1.05), emotional function (n=244; mean 0.81, SD 1.18), and adherence (n=103; mean 0.34, SD 0.53) were the most frequently assessed outcome domains. Self-developed PROMs were used in 21.4% (65/303) of the studies, and self-developed PREMs were used in 22.3% (68/303). PROMs (n=884) were assessed more frequently than PREMs (n=234). As the evidence level of the studies increased, the number of PROMs also increased (τ=−0.45), and the number of PREMs decreased (τ=0.35). Since 2000, not only has the number of studies using PROMs and PREMs increased, but the level of evidence and the number of outcome measurement instruments used have also increased, with the number of PREMs permanently remaining at a lower level.

Conclusions

There have been increasingly more studies, particularly high-evidence studies, which use PROMs and PREMs to evaluate telemedicine. PROMs have been used more frequently than PREMs. With the increasing maturity stage of telemedicine applications and higher evidence level, the use of PROMs increased in line with the recommendations of evaluation guidelines. Health-related quality of life and emotional function were measured in almost all the studies. Simultaneously, health literacy as a precondition for using the application adequately, alongside proper training and guidance, has rarely been reported. Further efforts should be pursued to standardize PROM and PREM collection in evaluation studies of telemedicine.

eHealth in Geriatric Rehabilitation: Systematic Review of Effectiveness, Feasibility, and Usability

BACKGROUND

eHealth has the potential to improve outcomes such as physical activity or balance in older adults receiving geriatric rehabilitation. However, several challenges such as scarce evidence on effectiveness, feasibility, and usability hinder the successful implementation of eHealth in geriatric rehabilitation.

OBJECTIVE

The aim of this systematic review was to assess evidence on the effectiveness, feasibility, and usability of eHealth interventions in older adults in geriatric rehabilitation.

METHODS

We searched 7 databases for randomized controlled trials, nonrandomized studies, quantitative descriptive studies, qualitative research, and mixed methods studies that applied eHealth interventions during geriatric rehabilitation. Included studies investigated a combination of effectiveness, usability, and feasibility of eHealth in older patients who received geriatric rehabilitation, with a mean age of ≥70 years. Quality was assessed using the Mixed Methods Appraisal Tool and a narrative synthesis was conducted using a harvest plot.

RESULTS

In total, 40 studies were selected, with clinical heterogeneity across studies. Of 40 studies, 15 studies (38%) found eHealth was at least as effective as non-eHealth interventions (56% of the 27 studies with a control group), 11 studies (41%) found eHealth interventions were more effective than non-eHealth interventions, and 1 study (4%) reported beneficial outcomes in favor of the non-eHealth interventions. Of 17 studies, 16 (94%) concluded that eHealth was feasible. However, high exclusion rates were reported in 7 studies of 40 (18%). Of 40 studies, 4 (10%) included outcomes related to usability and indicated that there were certain aging-related barriers to cognitive ability, physical ability, or perception, which led to difficulties in using eHealth.

CONCLUSIONS

eHealth can potentially improve rehabilitation outcomes for older patients receiving geriatric rehabilitation. Simple eHealth interventions were more likely to be feasible for older patients receiving geriatric rehabilitation, especially, in combination with another non-eHealth intervention. However, a lack of evidence on usability might hamper the implementation of eHealth. eHealth applications in geriatric rehabilitation show promise, but more research is required, including research with a focus on usability and participation.

Implementation and Evaluation of the Premature Infant Pain Profile-revised (PIPP-R) e-Learning Module for Assessing Pain in Infants

OBJECTIVES

The Premature Infant Pain Profile-revised (PIPP-R) is a well-established measure for infant pain assessment. The aim of this study was to evaluate the implementation and clinical utility of the PIPP-R electronic learning (e-Learning) module to promote standardized health care training for nurses.

MATERIALS AND METHODS

A descriptive mixed-methods study was conducted in 2 tertiary Neonatal Intensive Care Units in Canada. Nurses were recruited and asked to complete the PIPP-R e-Learning Module and evaluate it. A 26-item questionnaire was used to describe nurse demographics and clinical experience and to evaluate implementation success (ie, acceptability, feasibility, usability) and clinical utility.

RESULTS

In all, 98 nurses from 2 settings in Central and Eastern Canada participated; most were registered nurses highly experienced in neonatal nursing care. The majority had received previous training on the PIPP-R (61.2%) and routinely used it in practice (67.4%). They considered the e-Learning module as acceptable and feasible as it was easy to access (94.9%) and to navigate (94.8%). Content was considered clear (98.9%) and met users' learning needs (99.0%). Nurses agreed that completing the module improved their understanding of neonatal pain (96.0%) and was clinically useful in improving their ability to assess pain in neonates (97.9%). The module was accessed primarily from work settings (77.8%) using desktop computers (49.0%) or tablets (28.0%) and was usually completed in a single session (75.7%).

DISCUSSION

Nurses' evaluation of the PIPP-R e-Learning module was overwhelmingly positive. The module was perceived as easy to implement, clinically useful, and was considered as a promising online educational tool. Further testing in clinical practice is needed to build on the results of this study and support the importance of dissemination of this module for standardized training purposes.

Systematic development of a mobile preconception lifestyle programme for couples undergoing IVF: the PreLiFe-programme

STUDY QUESTION

Can we develop a preconception lifestyle programme for couples undergoing IVF that is in line with their needs.

SUMMARY ANSWER

A mobile preconception lifestyle programme was systematically developed based on expert opinion, literature and needs of IVF-patients.

WHAT IS KNOWN ALREADY

A healthy lifestyle prior to conception is not only beneficial for the general health of couples, but evidence on its importance for their reproductive health and the health of their children is also emerging. So far, the vast majority of fertility clinics do not offer a lifestyle programme for couples undergoing IVF. Therefore, the present study aimed to develop a lifestyle programme for IVF-couples.

STUDY DESIGN, SIZE, DURATION

The development of the PreLiFe-programme was guided by the steps of the Medical Research Council (MRC) framework for developing complex interventions, a systematic approach for developing theory- and evidence-based health promotion interventions.

PARTICIPANTS/MATERIALS, SETTINGS, METHODS

First, the evidence base on lifestyle programmes for IVF-couples was reviewed. Second, several iterations between an expert panel, the literature, and quantitative and qualitative data from IVF-patients identified the content, the format, behaviour change techniques and theory of the programme. Third, the PreLiFe-programme was produced and the expected process and outcomes of a randomized controlled trial assessing it were modelled. Finally, user tests among experts and patients and pilot tests among patients were conducted.

MAIN RESULTS AND ROLE OF CHANCE

The finally developed PreLiFe-programme is a mobile application to be used autonomously by both partners of IVF-couples during the first year of IVF, in combination with motivational interviewing over the telephone every three months (i.e. blended care). The PreLiFe-programme provides advice and skills training on physical activity, diet and mindfulness based stress reduction and is in part tailored based on monitoring and tracking the lifestyle of patients. Based on the literature the expert panel considers it plausible that all three components contribute to IVF-success rates. The PreLiFe-programme is likely to be acceptable to patients as it meets the need of patients for lifestyle advice and treatment information.

LIMITATIONS, REASON FOR CAUTION

The pilot in IVF-couples had a 3-month duration. The feasibility of the PreLiFe-programme in other infertile populations and/or upon longer use is yet to be examined. Whether the PreLiFe-programme effectively improves lifestyle and IVF-success rates is currently being examined in a trial randomizing heterosexual couples starting IVF to the PreLiFe-programme or an attention-control group for 12 months.

WIDER IMPLICATIONS OF THE FINDINGS

If the PreLiFe-programme improves lifestyle and the chance of IVF-success, it will be a powerful tool and provide guidance for implementing lifestyle programmes in fertility clinics.

STUDY FUNDING/COMPETING INTEREST(S)

Funded by the Research Foundation Flanders (FWO-TBM (Applied Biomedical Research with a Primary Social finality); reference: T005417N). The authors have no conflict of interest to report.

TRIAL REGISTRATION NUMBER

NCT03790449.

Usability, Acceptability, and Safety Analysis of a Computer-Tailored Web-Based Exercise Intervention (ExerciseGuide) for Individuals With Metastatic Prostate Cancer: Multi-Methods Laboratory-Based Study

BACKGROUND

Digital health interventions such as tailored websites are emerging as valuable tools to provide individualized exercise and behavioral change information for individuals diagnosed with cancer.

OBJECTIVE

The aim of this study is to investigate and iteratively refine the acceptability and usability of a web-based exercise intervention (ExerciseGuide) for men with metastatic prostate cancer and determine how well individuals can replicate the video-based exercise prescription.

METHODS

A laboratory-based multi-methods design was used, incorporating questionnaires, think-aloud tests, interviews, and movement screening among 11 men aged 63 to 82 years with metastatic prostate cancer. Overall, 9 participants were undergoing androgen deprivation therapy, and 2 were completing chemotherapy. Data were collected in two waves, with changes made for quality improvement after participant 5.

RESULTS

The intervention's usability score was deemed moderate overall but improved after modifications (from 60, SD 2.9 to 69.6, SD 2.2 out of 100). Overall, the participants found the intervention acceptable, with scores improving from wave 1 (24.2, SD 1.1 out of 30) to wave 2 (26.3, SD 2.1 out of 30). The personalized multimodal exercise prescription and computer-tailored education were seen as valuable. After wave 1, website navigation videos were added, medical terminology was simplified, and a telehealth component was included after expert real-time telehealth support was requested. Wave 2 changes included the added variety for aerobic exercise modes, reduced computer-tailoring question loads, and improved consistency of style and grammar. Finally, the participants could replicate the resistance exercise videos to a satisfactory level as judged by the movement screen; however, additional technique cueing within the videos is recommended to address safety concerns.

CONCLUSIONS

The acceptability and usability of ExerciseGuide were deemed satisfactory. Various problems were identified and resolved. Notably, the participants requested the inclusion of personalized expert support through telehealth. The resistance training algorithms were shown to provide appropriate content safely, and the users could replicate the exercise technique unaided to a satisfactory level. This study has optimized the ExerciseGuide intervention for further investigation in this population.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry (ANZCTR) ACTRN12618001978257; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12618001978257.

Conceptualizing Usability for the eHealth Context: Content Analysis of Usability Problems of eHealth Applications

BACKGROUND

Usability tests can be either formative (where the aim is to detect usability problems) or summative (where the aim is to benchmark usability). There are ample formative methods that consider user characteristics and contexts (ie, cognitive walkthroughs, interviews, and verbal protocols). This is especially valuable for eHealth applications, as health conditions can influence user-system interactions. However, most summative usability tests do not consider eHealth-specific factors that could potentially affect the usability of a system. One of the reasons for this is the lack of fine-grained frameworks or models of usability factors that are unique to the eHealth domain.

OBJECTIVE

In this study, we aim to develop an ontology of usability problems, specifically for eHealth applications, with patients as primary end users.

METHODS

We analyzed 8 data sets containing the results of 8 formative usability tests for eHealth applications. These data sets contained 400 usability problems that could be used for analysis. Both inductive and deductive coding were used to create an ontology from 6 data sets, and 2 data sets were used to validate the framework by assessing the intercoder agreement.

RESULTS

We identified 8 main categories of usability factors, including basic system performance, task-technology fit, accessibility, interface design, navigation and structure, information and terminology, guidance and support, and satisfaction. These 8 categories contained a total of 21 factors: 14 general usability factors and 7 eHealth-specific factors. Cohen κ was calculated for 2 data sets on both the category and factor levels, and all Cohen κ values were between 0.62 and 0.67, which is acceptable. Descriptive analysis revealed that approximately 69.5% (278/400) of the usability problems can be considered as general usability factors and 30.5% (122/400) as eHealth-specific usability factors.

CONCLUSIONS

Our ontology provides a detailed overview of the usability factors for eHealth applications. Current usability benchmarking instruments include only a subset of the factors that emerged from our study and are therefore not fully suited for summative evaluations of eHealth applications. Our findings support the development of new usability benchmarking tools for the eHealth domain.

Use and usability of the dr. Bart app and its relation with health care utilisation and clinical outcomes in people with knee and/or hip osteoarthritis

Background

Self-management is of paramount importance in the non-surgical treatment of knee/hip osteoarthritis (OA). Modern technologies offer the possibility of 24/7 self-management support. We developed an e-self-management application (dr. Bart app) for people with knee/hip OA.

The aim of this study was to document the use and usability of the dr. Bart app and its relation with health care utilisation and clinical outcomes in people with knee/hip OA.

Methods

For this study we used backend data for the first 26 weeks of use by the intervention group (N = 214) of an RCT examining the effectiveness of the dr. Bart app. A central element of the dr. Bart app is that it proposes a selection of 72 preformulated goals for health behaviours based on the ‘tiny habits method’ (e.g. after lunch I rise 12 times from my chair to train my leg muscles). The usability of the app was measured using the System Usability Scale questionnaire (SUS), on a scale of 0–100. To assess the association between the intensity of use of the app and health care utilisation (i.e., consultations in primary or secondary health care) and clinical outcomes (i.e., self-management behaviour, physical activity, health-related quality of life, illness perceptions, symptoms, pain, activities of daily living) we calculated Spearman rank correlation coefficients.

Results

Of the 214 participants, 171 (80%) logged in at least once with 151 (71%) choosing at least one goal and 114 (53%) completing at least one goal during the 26 weeks. Of those who chose at least one goal, 56 participants (37%) continued to log in for up to 26 weeks, 12 (8%) continued to select new goals from the offered goals and 37 (25%) continued to complete goals. Preformulated goals in the themes of physical activity (e.g., performing an exercise from the exercises library in the app) and nutrition (e.g., ‘eat two pieces of fruit today’) were found to be most popular with users. The mean usability scores (standard deviation) at the three and six month follow-ups were 65.9 (16.9) and 64.5 (17.5), respectively. The vast majority of associations between the intensity of use of the dr. Bart app and target outcomes were weak at ρ < (−) 0.25.

Conclusions

More than one-third of people with knee/hip OA who started using the app, continued to use it up to 26 weeks, though usability could be improved. Patients appear to have preferences for goals related to physical activity and nutrition, rather than for goals related to vitality and education. We found weak/no associations between the intensity of use of the dr. Bart app and health care utilisation and clinical outcomes.

Trial registration

(21 September 2017): Dutch Trial Register (Trial Number NTR6693/NL6505)

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06440-1.

A hand exercise mobile app for people with rheumatoid arthritis in Turkey: design, development and usability study

To design and develop a smartphone application for a structured hand exercise programme for patients with rheumatoid arthritis (RA) in Turkey and to test its usability. We followed a two-stage process: (1) Design and Development and (2) Usability testing. In stage 1, we used a qualitative user-centered design approach. We conducted a focus group (8 therapists and people with RA) to discuss the content, features and design to produce a prototype of the application. In a second focus group session, the participants tested the prototype, provided feedback and further revisions were made. In stage 2, 17 participants with RA used the app for 4 to 6 weeks. The System Usability Scale and the adapted Usability, Satisfaction and Ease to Use Questionnaires were used to measure usability, ease of use. Semi-structured interviews were conducted to explore user experiences with the application with 17 participants. In stage 1, the following themes were identified from the focus groups (a) login techniques (b) self-monitoring (c) exercises, (d) exercise diary, (e) information, (f) behavioral change and encouragement (g) exercise adherence. In stage 2, 3 themes were determined from interviews: (a) learning and accuracy, (b) ease of use, (c) motivation and adherence. USE and SUS scores indicated that users reported a high level of usability, satisfaction and ease of use. A mobile app for hand exercise for people with RA was developed using a mixed-method and iterative design. Participants perceived the mobile app as easy to use with high levels of satisfaction.

Supporting deprescribing in hospitalised patients: formative usability testing of a computerised decision support tool

Background

Despite growing evidence that deprescribing can improve clinical outcomes, quality of life and reduce the likelihood of adverse drug events, the practice is not widespread, particularly in hospital settings. Clinical risk assessment tools, like the Drug Burden Index (DBI), can help prioritise patients for medication review and prioritise medications to deprescribe, but are not integrated within routine care. The aim of this study was to conduct formative usability testing of a computerised decision support (CDS) tool, based on DBI, to identify modifications required to the tool prior to trialling in practice.

Methods

Our CDS tool comprised a DBI MPage in the electronic medical record (clinical workspace) that facilitated review of a patient’s DBI and medication list, access to deprescribing resources, and the ability to deprescribe. Two rounds of scenario-based formative usability testing with think-aloud protocol were used. Seventeen end-users participated in the testing, including junior and senior doctors, and pharmacists.

Results

Participants expressed positive views about the DBI CDS tool but testing revealed a number of clear areas for improvement. These primarily related to terminology used (i.e. what is a DBI and how is it calculated?), and consistency of functionality and display. A key finding was that users wanted the CDS tool to look and function in a similar way to other decision support tools in the electronic medical record. Modifications were made to the CDS tool in response to user feedback.

Conclusion

Usability testing proved extremely useful for identifying components of our CDS tool that were confusing, difficult to locate or to understand. We recommend usability testing be adopted prior to implementation of any digital health intervention. We hope our revised CDS tool equips clinicians with the knowledge and confidence to consider discontinuation of inappropriate medications in routine care of hospitalised patients. In the next phase of our project, we plan to pilot test the tool in practice to evaluate its uptake and effectiveness in supporting deprescribing in routine hospital care.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01484-z.

Usability of a Fall Risk mHealth App for People With Multiple Sclerosis: Mixed Methods Study

BACKGROUND

Multiple sclerosis (MS) is a chronic, neurodegenerative disease that causes a range of motor, sensory, and cognitive symptoms. Due to these symptoms, people with MS are at a high risk for falls, fall-related injuries, and reductions in quality of life. There is no cure for MS, and managing symptoms and disease progression is important to maintain a high quality of life. Mobile health (mHealth) apps are commonly used by people with MS to help manage their health. However, there are limited health apps for people with MS designed to evaluate fall risk. A fall risk app can increase access to fall risk assessments and improve self-management. When designing mHealth apps, a user-centered approach is critical for improving use and adoption.

OBJECTIVE

The purpose of this study is to undergo a user-centered approach to test and refine the usability of the app through an iterative design process.

METHODS

The fall risk app Steady-MS is an extension of Steady, a fall risk app for older adults. Steady-MS consists of 2 components: a 25-item questionnaire about demographics and MS symptoms and 5 standing balance tasks. Data from the questionnaire and balance tasks were inputted into an algorithm to compute a fall risk score. Two iterations of semistructured interviews (n=5 participants per iteration) were performed to evaluate usability. People with MS used Steady-MS on a smartphone, thinking out loud. Interviews were recorded, transcribed, and developed into codes and themes. People with MS also completed the System Usability Scale.

RESULTS

A total of 3 themes were identified: intuitive navigation, efficiency of use, and perceived value. Overall, the participants found Steady-MS efficient to use and useful to learn their fall risk score. There were challenges related to cognitive overload during the balance tasks. Modifications were made, and after the second iteration, people with MS reported that the app was intuitive and efficient. Average System Usability Scale scores were 95.5 in both iterations, representing excellent usability.

CONCLUSIONS

Steady-MS is the first mHealth app for people with MS to assess their overall risk of falling and is usable by a subset of people with MS. People with MS found Steady-MS to be usable and useful for understanding their risk of falling. When developing future mHealth apps for people with MS, it is important to prevent cognitive overload through simple and clear instructions and present scores that are understood and interpreted correctly through visuals and text. These findings underscore the importance of user-centered design and provide a foundation for the future development of tools to assess and prevent scalable falls for people with MS. Future steps include understanding the validity of the fall risk algorithm and evaluating the clinical utility of the app.

Patients' compliance behavior in a personalized mobile patient education system (PMPES) setting: Rational, social, or personal choices?

OBJECTIVE

With the advancement of mobile technologies, patients can access medical and patient educational information anytime and anywhere. Computer-aided patient education has been advocated as a key means of interventions for improving patient knowledge and compliance (i.e., adherence). However, evidence of the efficacy of computer-aided patient education remains relatively limited. For example, little is known about how the latest mobile technologies influence patients' compliance intention and their actual compliance behavior. The objective of this study is to investigate patients' compliance intention and behavior using a personalized mobile patient education system (PMPES) as a novel technological intervention for patients based on rational choice theory (RCT) and the theory of planned behavior (TPB).

MATERIALS AND METHODS

We conducted a field survey with 125 actual patients in U.S. who obtained their patient education through PMPES while seeking medical treatment advice from their doctors. We used partial least squares (PLS) regression path modeling to test our model.

RESULTS

We found that, based on RCT, the benefits of compliance and cost/threat of noncompliance positively influenced intention toward treatment compliance; in contrast, costs of compliance negatively influenced intention toward treatment compliance. However, the benefits of noncompliance had no effect on intention toward treatment compliance. The results also indicated that intention toward treatment compliance, response efficacy, and self-efficacy related to TPB jointly influenced the degree of actual compliance behaviors. Social influence factors including subjective norms and descriptive norms had no influence on patients' actual treatment compliance behavior.

CONCLUSION

Overall, the research model explains 69.2 % of the variance in patients' actual compliance behavior. We find our model robust in using RCT as a key theoretical lens for the assessment of patients' compliance intention to follow medical recommendations enabled by the PMPES and delivered to mobile devices. The factors associated with RCT and TPB jointly influence patients' actual compliance behavior. Future mobile patient education programs should consider patients' age groups, mixed-gender groups, different medical settings, and cross-cultural contexts.

Contextual Structured Reporting in Radiology: Implementation and Long-Term Evaluation in Improving the Communication of Critical Findings

Structured reporting contributes to the completeness of radiology reports and improves quality. Both the content and the structure are essential for successful implementation of structured reporting. Contextual structured reporting is tailored to a specific scenario and can contain information retrieved from the context. Critical findings detected by imaging need urgent communication to the referring physician. According to guidelines, the occurrence of this communication should be documented in the radiology reports and should contain when, to whom and how was communicated. In free-text reporting, one or more of these required items might be omitted. We developed a contextual structured reporting template to ensure complete documentation of the communication of critical findings. The WHEN and HOW items were included automatically, and the insertion of the WHO-item was facilitated by the template. A pre- and post-implementation study demonstrated a substantial improvement in guideline adherence. The template usage improved in the long-term post-implementation study compared with the short-term results. The two most often occurring categories of critical findings are “infection / inflammation” and “oncology”, corresponding to the a large part of urgency level 2 (to be reported within 6 h) and level 3 (to be reported within 6 days), respectively. We conclude that contextual structured reporting is feasible for required elements in radiology reporting and for automated insertion of context-dependent data. Contextual structured reporting improves guideline adherence for communication of critical findings.