Extending cBioPortal for Therapy Recommendation Documentation in Molecular Tumor Boards: Development and Usability Study

BACKGROUND

In molecular tumor boards (MTBs), patients with rare or advanced cancers are discussed by a multidisciplinary team of health care professionals. Software support for MTBs is lacking; in particular, tools for preparing and documenting MTB therapy recommendations need to be developed.

OBJECTIVE

We aimed to implement an extension to cBioPortal to provide a tool for the documentation of therapy recommendations from MTB sessions in a secure and standardized manner. The developed extension should be embedded in the patient view of cBioPortal to enable easy documentation during MTB sessions. The resulting architecture for storing therapy recommendations should be integrable into various hospital information systems.

METHODS

On the basis of a requirements analysis and technology analysis for authentication techniques, a prototype was developed and iteratively refined through a user-centered development process. In conclusion, the tool was evaluated via a usability evaluation, including interviews, structured questionnaires, and the System Usability Scale.

RESULTS

The patient view of cBioPortal was extended with a new tab that enables users to document MTB sessions and therapy recommendations. The role-based access control was expanded to allow for a finer distinction among the rights to view, edit, and delete data. The usability evaluation showed overall good usability and a System Usability Scale score of 83.57.

CONCLUSIONS

This study demonstrates how cBioPortal can be extended to not only visualize MTB patient data but also be used as a documentation platform for therapy recommendations.

Medication Safety Amid Technological Change: Usability Evaluation to Inform Inpatient Nurses' Electronic Health Record System Transition

BACKGROUND

Electronic health record (EHR) system transitions are challenging for healthcare organizations. High-volume, safety-critical tasks like barcode medication administration (BCMA) should be evaluated, yet standards for ensuring safety during transition have not been established.

OBJECTIVE

Identify risks in common and problem-prone medication tasks to inform safe transition between BCMA systems and establish benchmarks for future system changes.

DESIGN

Staff nurses completed simulation-based usability testing in the legacy system (R1) and new system pre- (R2) and post-go-live (R3). Tasks included (1) Hold/Administer, (2) IV Fluids, (3) PRN Pain, (4) Insulin, (5) Downtime/PRN, and (6) Messaging. Audiovisual recordings of task performance were systematically analyzed for time, navigation, and errors. The System Usability Scale measured perceived usability and satisfaction. Post-simulation interviews captured nurses' qualitative comments and perceptions of the systems.

PARTICIPANTS

Fifteen staff nurses completed 2-3-h simulation sessions. Eleven completed both R1 and R2, and seven completed all three rounds. Clinical experience ranged from novice (< 1 year) to experienced (> 10 years). Practice settings included adult and pediatric patient populations in ICU, stepdown, and acute care departments.

MAIN MEASURES

Task completion rates/times, safety and non-safety-related use errors (interaction difficulties), and user satisfaction.

KEY RESULTS

Overall success rates remained relatively stable in all tasks except two: IV Fluids task success increased substantially (R1: 17%, R2: 54%, R3: 100%) and Downtime/PRN task success decreased (R1: 92%, R2: 64%, R3: 22%). Among the seven nurses who completed all rounds, overall safety-related errors decreased 53% from R1 to R3 and 50% from R2 to R3, and average task times for successfully completed tasks decreased 22% from R1 to R3 and 38% from R2 to R3.

CONCLUSIONS

Usability testing is a reasonable approach to compare different BCMA tasks to anticipate transition problems and establish benchmarks with which to monitor and evaluate system changes going forward.

Usability Evaluation of a Knowledge Graph-Based Dementia Care Intelligent Recommender System: Mixed Methods Study

BACKGROUND

Knowledge graph-based recommender systems offer the possibility of meeting the personalized needs of people with dementia and their caregivers. However, the usability of such a recommender system remains unknown.

OBJECTIVE

This study aimed to evaluate the usability of a knowledge graph-based dementia care intelligent recommender system (DCIRS).

METHODS

We used a convergent mixed methods design to conduct the usability evaluation, including the collection of quantitative and qualitative data. Participants were recruited through social media advertisements. After 2 weeks of DCIRS use, feedback was collected with the Computer System Usability Questionnaire and semistructured interviews. Descriptive statistics were used to describe sociodemographic characteristics and questionnaire scores. Qualitative data were analyzed systematically using inductive thematic analysis.

RESULTS

A total of 56 caregivers were recruited. Quantitative data suggested that the DCIRS was easy for caregivers to use, and the mean questionnaire score was 2.14. Qualitative data showed that caregivers generally believed that the content of the DCIRS was professional, easy to understand, and instructive, and could meet users' personalized needs; they were willing to continue to use it. However, the DCIRS also had some shortcomings. Functions that enable interactions between professionals and caregivers and that provide caregiver support and resource recommendations might be added to improve the system's usability.

CONCLUSIONS

The recommender system provides a solution to meet the personalized needs of people with dementia and their caregivers and has the potential to substantially improve health outcomes. The next step will be to optimize and update the recommender system based on caregivers' suggestions and evaluate the effect of the application.

Exploring usability problems of mHealth applications developed for cervical cancer: An empirical study

Objectives

Nowadays, mobile health applications are developed to raise awareness and facilitate screening and treatment of cervical cancer, while a very few studies have been conducted focusing on the measurement and assurance of usability and exploring the acceptable user experience of such applications. Usability issues become a crucial concern for such cervical-cancer-related applications because users with diverse backgrounds in terms of education, information technology literacy, and geographic reasons are required to access those applications. The objective of this research is to evaluate the usability of mobile health applications developed for cervical cancer patients.

Methods

Two evaluation studies were conducted following the expert evaluation and a questionnaire-based user study. A total of four cervical-cancer-related applications that are focusing on the Awareness and Diagnosis theme were selected and each of the applications was evaluated by four usability experts. Then, a user study (n = 80) based on the Goal Question Metric was conducted to reveal the usability problems of four selected applications. Finally, findings of both evaluations were aggregated and analyzed.

Results

Both approaches showed that all applications suffer from several usability problems while "Cervical Cancer Guide" performs better and "Cervical Cancer Tracker" showed the least in performance from the usability perspective. Again, the Goal Question Metric performs noticeably better in assessing the learnability of the applications, while the analytical heuristic evaluation performs better in identifying the issues that cause user annoyance.

Conclusion

The methodology adopted and the usability problems revealed through this study can be well utilized by the information technology professionals or user interface designers for designing, evaluating, and developing the cervical-cancer-related applications with enhanced usability and user experience.

Learning Designers as Expert Evaluators of Usability: Understanding Their Potential Contribution to Improving the Universality of Interface Design for Health Resources

User-based evaluation by end users is an essential step in designing useful interfaces. Inspection methods can offer an alternate approach when end-user recruitment is problematic. A Learning Designers' usability scholarship could offer usability evaluation expertise adjunct to multidisciplinary teams in academic settings. The feasibility of Learning Designers as 'expert evaluators' is assessed within this study. Two groups, healthcare professionals and Learning Designers, applied a hybrid evaluation method to generate usability feedback from a palliative care toolkit prototype. Expert data were compared to end-user errors detected from usability testing. Interface errors were categorised, meta-aggregated and severity calculated. The analysis found that reviewers detected N = 333 errors, with N = 167 uniquely occurring within the interface. Learning Designers identified errors at greater frequencies (60.66% total interface errors, mean (M) = 28.86 per expert) than other evaluator groups (healthcare professionals 23.12%, M = 19.25 and end users 16.22%, M = 9.0). Patterns in severity and error types were also observed between reviewer groups. The findings suggest that Learning Designers are skilled in detecting interface errors, which benefits developers assessing usability when access to end users is limited. Whilst not offering rich narrative feedback generated by user-based evaluations, Learning Designers complement healthcare professionals' content-specific knowledge as a 'composite expert reviewer' with the ability to generate meaningful feedback to shape digital health interfaces.

Design and validation of a novel online platform to support the usability evaluation of wearable robotic devices

Wearable robotic devices (WRD) are still struggling to fulfill their vast potential. Inadequate daily life usability is one of the main hindrances to increased technology acceptance. Improving usability evaluation practices during the development of WRD could help address these limitations. In this work, we present the design and validation of a novel online platform aiming to fill this gap, the Interactive Usability Toolbox (IUT). This platform consists of a public website that offers an interactive, context-specific search within a database of 154 user research methods and educational information about usability. In a dedicated study, the effect of this platform to support usability evaluation was investigated. Twelve WRD experts were asked to complete the task of defining usability evaluation protocols for two specific use cases. The platform was provided to support one of the use cases. The quality and composition of the proposed protocols were assessed by (i) two blinded reviewers, (ii) the participants themselves, and (iii) the study coordinators. We showed that using the IUT significantly affected the proposed evaluation focus, shifting protocols from mainly effectiveness-oriented to more user-focused studies. The protocol quality, as rated by the external reviewers, remained equivalent to those designed with conventional strategies. A mixed-method usability evaluation of the platform yielded an overall positive image, with detailed suggestions for further improvements. The IUT is expected to positively affect the evaluation and development of WRD through its educational value, the context-specific recommendations supporting ongoing benchmarking endeavors, and highlighting the value of qualitative user research.

Usability Evaluation of the Preoperative ISBAR (Identification, Situation, Background, Assessment, and Recommendation) Desktop Virtual Reality Application: Qualitative Observational Study

BACKGROUND

Systematic communication, such as the ISBAR (identification, situation, background, assessment, recommendation) approach, comprises a generic, transferable nontechnical skill. It can be used during the handover of patients set to undergo surgery and can be practiced in various ways, including virtual reality (VR). VR increasingly has been implemented and valued in nursing education as a positive contribution to teach students about pre- and postoperative nursing. A new nonimmersive 3D learning activity called the Preoperative ISBAR Desktop VR Application has been developed for undergraduate nursing students to learn preoperative handover using the ISBAR approach. However, the usability of this learning activity has not been studied.

OBJECTIVE

This study aimed to investigate how second-year undergraduate nursing students evaluated the usability of the Preoperative ISBAR Desktop VR Application.

METHODS

This was a qualitative study with observation and interviews. The inclusion criteria were undergraduate second-year nursing students of varying ages, gender, and anticipated technological competence. The System Usability Scale (SUS) questionnaire was used to get a score on overall usability.

RESULTS

A total of 9 second-year nursing students aged 22-29 years participated in the study. The average score on the SUS was 83 (range 0-100), which equals a "B" on the graded scale and is excellent for an adjective-grade rating. The students expressed increased motivation to learn while working in self-instructed desktop VR. Still, a few technical difficulties occurred, and some students reported that they experienced some problems comprehending the instructions provided in the application. Long written instructions and a lack of self-pacing built into the application were considered limitations.

CONCLUSIONS

The nursing students found the application to be usable overall, giving it an excellent usability score and noting that the application provided opportunities for active participation, which was motivational and facilitated their perceived learning outcomes. The next version of the application, to be used in a randomized controlled trial, will be upgraded to address technological and comprehension issues.

Improving Tractor Safety: A Comparison between the Usability of a Conventional and Enhanced Rear-Mounted Foldable ROPS (FROPS)

Tractor rollover is the main cause of both fatal and non-fatal injuries in agriculture. The foldable rollover protective structure (FROPS) can help to prevent these injuries. However, in many cases, the FROPS is left in a folded-down position. Human factor and rear-mounted FROPS technical characteristics influence operators' behavior in roll-bar handling. To improve the FROPS's comfortable use, the prototype of an enhanced handling system was developed, and its usability was tested and compared with a conventional FROPS. Usability was assessed with 93 novice tractor users through an ad hoc questionnaire (investigating perceived effort, time demand, the posture adopted and satisfaction) and observations (investigating effectiveness and efficiency) during lowering and raising tasks. For both tasks, the participants perceived significantly less effort, less physical discomfort, a higher level of satisfaction and less time demand while operating the enhanced FROPS. Observations showed that the critical issues that emerged for the conventional FROPS were eliminated by adopting the developed and implemented handling system. The developed handling system showed itself to be usable and effective in making the FROPS easier and safer to be operated.

Designing a Social Robot Companion to Support Homecare: Usability Results

Earlier studies show frail seniors often experience loneliness and depression. Moreover, frailty can lead problems with medication and nutrition patterns. The availability of family care and/or nursing care at home is limited. Digital companions, such as social robots, could complement homecare nurses, thereby improving the quality of care to frail seniors. The Guardian project has co- designed with end-users, a social robot providing social company and health support. To assess the digital and co-created solution, usability evaluations have been conducted with 43 participants distributed as fairly as possible between frail seniors, family carers and professional nurses; in three different European areas: The Netherlands, Italy and Switzerland. The goal of this study is to evaluate if the GUARDIAN solution is accepted by the target users and also gather data on how to improve the system for ensuring added-value in home care. The iterative method based on user-centered approach put the end-users at the centre of the usability evaluation. Through thematic analysis of the qualitative datasets, we conclude that a high number of users accept the solution and describe it as useful. End-user needs have been mainly addressed but some new improvements have been pointed out by the participants and some other needs have been uncovered.

Parental Knowledge, Self-confidence, and Usability Evaluation of a Web-Based Infographic for Pediatric Concussion: Multimethod Study

BACKGROUND

Concussions, which are known as mild traumatic brain injuries, are complex injuries caused by direct or indirect blows to the head and are increasingly being recognized as a significant public health concern for children and their families. Previous research has identified few studies examining the efficacy of educational interventions on parental concussion knowledge. The aim of this research was to actively work together with children who have experienced a concussion and their parents to develop, refine, and evaluate the usability of a web-based infographic for pediatric concussion.

OBJECTIVE

The objective of this study was to report on the usability of the infographic, parental knowledge, and self-confidence in pediatric concussion knowledge before and after exposure to the infographic.

METHODS

A multiphase, multimethod research design using patient engagement techniques was used to develop a web-based infographic. For this phase of the research (usability, knowledge, and confidence evaluation), parents who could communicate in English were recruited via social media platforms and invited to complete web-based questionnaires. Electronic preintervention and postintervention questionnaires were administered to parents to assess changes to concussion knowledge and confidence after viewing the infographic. A usability questionnaire with 11 items was also completed.

RESULTS

A web-based, infographic was developed. The infographic is intended for parents and children and incorporates information that parents and children identified as both wants and needs about concussion alongside the best available research evidence on pediatric concussion. A total of 31 surveys were completed by parents. The mean scores for each item on the usability surveys ranged from 8.03 (SD 1.70) to 9.26 (SD 1.09) on a 10-point Likert scale, indicating that the usability components of the infographic were largely positive. There was no statistically significant difference between preintervention and postintervention knowledge scores (Z=-0.593; P=.55; both preintervention and postintervention knowledge scores had a median of 9 out of 10). In contrast, there was a statistically significant difference between preintervention (mean 3.9/5, SD 0.56) and postintervention (mean 4.4/5, SD 0.44) confidence in knowledge scores (t₃₀=-5.083; P<.001).

CONCLUSIONS

Our results demonstrate that parents positively rated a web-based, infographic for pediatric concussion. In addition, although there was no statistically significant difference overall in parents' knowledge scores before and after viewing the infographic, their confidence in their knowledge did significantly increase. These results suggest that using a web-based infographic as a knowledge translation intervention may be useful in increasing parents' confidence in managing their child's concussion.

Intention Detection Strategies for Robotic Upper-Limb Orthoses: A Scoping Review Considering Usability, Daily Life Application, and User Evaluation

Wearable robotic upper limb orthoses (ULO) are promising tools to assist or enhance the upper-limb function of their users. While the functionality of these devices has continuously increased, the robust and reliable detection of the user's intention to control the available degrees of freedom remains a major challenge and a barrier for acceptance. As the information interface between device and user, the intention detection strategy (IDS) has a crucial impact on the usability of the overall device. Yet, this aspect and the impact it has on the device usability is only rarely evaluated with respect to the context of use of ULO. A scoping literature review was conducted to identify non-invasive IDS applied to ULO that have been evaluated with human participants, with a specific focus on evaluation methods and findings related to functionality and usability and their appropriateness for specific contexts of use in daily life. A total of 93 studies were identified, describing 29 different IDS that are summarized and classified according to a four-level classification scheme. The predominant user input signal associated with the described IDS was electromyography (35.6%), followed by manual triggers such as buttons, touchscreens or joysticks (16.7%), as well as isometric force generated by residual movement in upper-limb segments (15.1%). We identify and discuss the strengths and weaknesses of IDS with respect to specific contexts of use and highlight a trade-off between performance and complexity in selecting an optimal IDS. Investigating evaluation practices to study the usability of IDS, the included studies revealed that, primarily, objective and quantitative usability attributes related to effectiveness or efficiency were assessed. Further, it underlined the lack of a systematic way to determine whether the usability of an IDS is sufficiently high to be appropriate for use in daily life applications. This work highlights the importance of a user- and application-specific selection and evaluation of non-invasive IDS for ULO. For technology developers in the field, it further provides recommendations on the selection process of IDS as well as to the design of corresponding evaluation protocols.

Utilizing a Human-Computer Interaction Approach to Evaluate the Design of Current Pharmacogenomics Clinical Decision Support

A formal assessment of pharmacogenomics clinical decision support (PGx-CDS) by providers is lacking in the literature. The objective of this study was to evaluate the usability of PGx-CDS tools that have been implemented in a healthcare setting. We enrolled ten prescribing healthcare providers and had them complete a 60-min usability session, which included interacting with two PGx-CDS scenarios using the “Think Aloud” technique, as well as completing the Computer System Usability Questionnaire (CSUQ). Providers reported positive comments, negative comments, and suggestions for the two PGx-CDS during the usability testing. Most provider comments were in favor of the current PGx-CDS design, with the exception of how the genotype and phenotype information is displayed. The mean CSUQ score for the PGx-CDS overall satisfaction was 6.3 ± 0.95, with seven strongly agreeing and one strongly disagreeing for overall satisfaction. The implemented PGx-CDS at our institution was well received by prescribing healthcare providers. The feedback collected from the session will guide future PGx-CDS designs for our healthcare system and provide a framework for other institutions implementing PGx-CDS.

Methodological Quality of User-Centered Usability Evaluation of Ambient Assisted Living Solutions: A Systematic Literature Review

This study aimed to determine the methodological quality of user-centered usability evaluation of Ambient Assisted Living (AAL) solutions by (i) identifying the characteristics of the AAL studies reporting on user-centered usability evaluation, (ii) systematizing the methods, procedures and instruments being used, and (iii) verifying if there is evidence of a common understanding on methods, procedures, and instruments for user-centered usability evaluation. An electronic search was conducted on Web of Science, Scopus, and IEEE Xplore databases, combining relevant keywords. Then, titles and abstracts were screened against inclusion and exclusion criteria, and the full texts of the eligible studies were retrieved and screened for inclusion. A total of 44 studies were included. The results show a great heterogeneity of methods, procedures, and instruments to evaluate the usability of AAL solutions and, in general, the researchers fail to consider and report relevant methodological aspects. Guidelines and instruments to assess the quality of the studies might help improving the experimental design and reporting of studies on user-centered usability evaluation of AAL solutions.

Toward a Conversational Agent to Support the Self-Management of Adults and Young Adults With Sickle Cell Disease: Usability and Usefulness Study

Sickle cell disease (SCD) is the most common genetic blood disorder in the world and affects millions of people. With aging, patients encounter an increasing number of comorbidities that can be acute, chronic, and potentially lethal (e.g., pain, multiple organ damages, lung disease). Comprehensive and preventive care for adults with SCD faces disparities (e.g., shortage of well-trained providers). Consequently, many patients do not receive adequate treatment, as outlined by evidence-based guidelines, and suffer from mistrust, stigmatization or neglect. Thus, adult patients often avoid necessary care, seek treatment only as a last resort, and rely on self-management to maintain control over the course of the disease. Hopefully, self-management positively impacts health outcomes. However, few patients possess the required skills (e.g., disease-specific knowledge, self-efficacy), and many lack motivation for effective self-care. Health coaching has emerged as a new approach to enhance patients' self-management and support health behavior changes. Recent studies have demonstrated that conversational agents (chatbots) could effectively support chronic patients' self-management needs, improve self-efficacy, encourage behavior changes, and reduce disease-severity. To date, the use of chatbots to support SCD self-management remains largely under-researched. Consequently, we developed a high-fidelity prototype of a fully automated health coaching chatbot, following patient-important requirements and preferences collected during our previous work. We recruited a small convenience sample of adults with SCD to examine the usability and perceived usefulness of the system. Participants completed a post-test survey using the System Usability Scale and the Usefulness Scale for Patient Information Material questionnaire. Thirty-three patients participated. The majority (64%) was affected by the most clinically severe SCD genotypes (Hb SS, HbSβ0). Most participants (94%) rated the chatbots as easy and fun to use, while 88% perceived it as useful support for patient empowerment. In the qualitative phase, 72% of participants expressed their enthusiasm using the chatbot, and 82% emphasized its ability to improve their knowledge about self-management. Findings suggest that chatbots could be used to promote the acquisition of recommended health behaviors and self-care practices related to the prevention of the main symptoms of SCD. Further work is needed to refine the system, and to assess clinical validity.

Usability of a Co-designed eHealth Prototype for Caregivers: Combination Study of Three Frameworks

Background

Co-design (or the participation of users) has shown great potential in the eHealth domain, demonstrating positive results. Nevertheless, the co-design approach cannot guarantee the usability of the system designed, and usability assessment is a complex analysis to perform, as evaluation criteria will differ depending on the usability framework (or set of criteria) used. ISO (International Organization for Standardization) on usability (ISO 9241-210), Nielsen heuristic, and Garrett element of user experience inform different yet complementary aspects of usability.

Objective

This study aims to assess the usability and user experience of a co-design prototype by combining 3 complementary frameworks.

Methods

To help caregivers provide care for functionally impaired older people, an eHealth tool was co-designed with caregivers, health and social service professionals, and community workers assisting caregivers. The prototype was a website that aims to support the help-seeking process for caregivers (finding resources) and allow service providers to advertise their services (offering resources). We chose an exploratory study method to assess usability in terms of each objective. The first step was to assess users’ first impressions of the website. The second was a task scenario with a think-aloud protocol. The final step was a semistructured interview. All steps were performed individually (with a moderator) in a single session. The data were analyzed using 3 frameworks.

Results

A total of 10 participants were recruited, 5 for each objective of the website. We were able to identify several usability problems, most of which were located in the information design and interface design dimensions (Garrett framework). Problems in both dimensions were mainly coded as effectiveness and efficiency (ISO framework) and error prevention and match between the systemand the real world (Nielsen heuristic).

Conclusions

Our study provided a novel contribution about usability analysis by combining the 3 different models to classify the problems found. This combination provided a holistic understanding of the usability improvements needed. It can also be used to analyze other eHealth products.

International Registered Report Identifier (IRRID)

RR2-10.2196/11634

Usability Evaluation of the mLab App for Improving Home HIV Testing Behaviors in Youth at Risk of HIV Infection

Improving access to HIV testing among youth at high risk is essential for reaching those who are most at risk for HIV and least likely to access health care services. This study evaluates the usability of mLab, an app with image-processing feature that analyzes photos of OraQuick HIV self-tests and provides real-time, personalized feedback. mLab includes HIV prevention information, testing reminders, and instructions. It was developed through iterative feedback with a youth advisory board (N = 8). The final design underwent heuristic (N = 5) and end-user testing (N = 20). Experts rated mLab following Nielsen's heuristic checklist. End-users used the Health Information Technology Usability Evaluation Scale. While there were some usability problems, overall study participants found mLab useful and user-friendly. This study provides important insights into using a mobile app with imaging for interpreting HIV test results with the goal of improving HIV testing and prevention in populations at high risk.

Usability Evaluation of a Cognitive-Behavioral App-Based Intervention for Binge Eating and Related Psychopathology: A Qualitative Study

Despite their promise as a scalable intervention modality for binge eating and related problems, reviews show that engagement of app-based interventions is variable. Issues with usability may account for this. App developers should undertake usability testing so that any problems can be identified and fixed prior to dissemination. We conducted a qualitative usability evaluation of a newly-developed app for binge eating in 14 individuals with a diagnostic- or subthreshold-level binge eating symptoms. Participants completed a semi-structured interview and self-report measures. Qualitative data were organized into six themes: usability, visual design, user engagement, content, therapeutic persuasiveness, and therapeutic alliance. Qualitative and quantitative results indicated that the app demonstrated good usability. Key advantages reported were its flexible content-delivery formats, level of interactivity, easy-to-understand information, and ability to track progress. Concerns with visual aesthetics and lack of professional feedback were raised. Findings will inform the optimal design of app-based interventions for eating disorder symptoms.

Implementation of a Computerized Provider Order Entry System in a Pediatric Hospital in Canada

The Centre Hospitalier Universitaire Sainte-Justine (Montreal, Canada) is a pediatric academic tertiary hospital that has begun the implementation of a commercial computerized provider order entry system (CPOE) in October 2019. The objectives of this paper are 1) to estimate the impact of the CPOE system on medication errors, and 2) to identify vulnerability issues related to the configuration of the CPOE system's design. Using a pre-post implementation methodology measuring medication errors captured by clinical pharmacists revealed that the implementation of a CPOE has eliminated all prescription conformity (e.g., missing fields) and legibility errors. Pharmacists have continued to detect medication errors, especially inappropriate dosing instructions, and to intervene in similar clinical situations (medication reconciliation, deprescribing, adjusting orders). Additionally, the vulnerability analysis, based on typical clinical order test cases in an inpatient pediatric setting, highlighted the need to configure a clinical decision support system that can identify inappropriate dosing instructions for pediatric patients.

Comparing Single-Page, Multipage, and Conversational Digital Forms in Health Care: Usability Study

Background

Even in the era of digital technology, several hospitals still rely on paper-based forms for data entry for patient admission, triage, drug prescriptions, and procedures. Paper-based forms can be quick and convenient to complete but often at the expense of data quality, completeness, sustainability, and automated data analytics. Digital forms can improve data quality by assisting the user when deciding on the appropriate response to certain data inputs (eg, classifying symptoms). Greater data quality via digital form completion not only helps with auditing, service improvement, and patient record keeping but also helps with novel data science and machine learning research. Although digital forms are becoming more prevalent in health care, there is a lack of empirical best practices and guidelines for their design. The study-based hospital had a definite plan to abolish the paper form; hence, it was not necessary to compare the digital forms with the paper form.

Objective

This study aims to assess the usability of three different interactive forms: a single-page digital form (in which all data input is required on one web page), a multipage digital form, and a conversational digital form (a chatbot).

Methods

The three digital forms were developed as candidates to replace the current paper-based form used to record patient referrals to an interventional cardiology department (Cath-Lab) at Altnagelvin Hospital. We recorded usability data in a counterbalanced usability test (60 usability tests: 20 subjects×3 form usability tests). The usability data included task completion times, System Usability Scale (SUS) scores, User Experience Questionnaire data, and data from a postexperiment questionnaire.

Results

We found that the single-page form outperformed the other two digital forms in almost all usability metrics. The mean SUS score for the single-page form was 76 (SD 15.8; P=.01) when compared with the multipage form, which had a mean score of 67 (SD 17), and the conversational form attained the lowest scores in usability testing and was the least preferred choice of users, with a mean score of 57 (SD 24). An SUS score of >68 was considered above average. The single-page form achieved the least task completion time compared with the other two digital form styles.

Conclusions

In conclusion, the digital single-page form outperformed the other two forms in almost all usability metrics; it had the least task completion time compared with those of the other two digital forms. Moreover, on answering the open-ended question from the final customized postexperiment questionnaire, the single-page form was the preferred choice.

Development and usability evaluation of VOICES: A digital health tool to identify elder mistreatment

BACKGROUND/OBJECTIVES

A major barrier for society in overcoming elder mistreatment is an inability to accurately identify victims. There are several barriers to self-reporting elder mistreatment, including fear of nursing home placement or losing autonomy or a caregiver. Existing strategies to identify elder mistreatment neglect to empower those who experience it with tools for self-reporting. In this project, we developed and evaluated the usability of VOICES, a self-administrated digital health tool that screens, educates, and motivates older adults to self-report elder mistreatment.

DESIGN

Cross-sectional study with User-Centered Design (UCD) approach.

SETTING

Yale School of Medicine and the Agency on Aging of South-Central Connecticut.

PARTICIPANTS

Thirty eight community-dwelling and cognitively intact older adults aged 60 years and older, caregivers, clinicians, and social workers.

INTERVENTION

A tablet-based self-administrated digital health tool that screens, educates, and motivates older adults to self-report elder mistreatment.

MEASUREMENTS

Qualitative and quantitative data were obtained from: (1) focus groups participants including: feedback from open-ended discussion, demographics, and a post-session survey; (2) usability evaluation including: demographics, usability measures, comfortability with technology, emotional state, and open-ended feedback.

RESULTS

Focus group participants (n = 24) generally favored using a tablet-based tool to screen for elder mistreatment and expressed comfort answering questions on elder mistreatment using tablets. Usability evaluation participants (n = 14) overall scored VOICES a mean System Usability Scale (SUS) score of 86.6 (median = 88.8), higher than the benchmark SUS score of 68, indicating excellent ease of use. In addition, 93% stated that they would recommend the VOICES tool to others and 100% indicated understanding of VOICES' information and content.

CONCLUSION

Our findings show that older adults are capable, willing, and comfortable with using the innovative and self-administrated digital tool for elder mistreatment screening. Our future plan is to conduct a feasibility study to evaluate the use of VOICES in identifying suspicion of mistreatment.

Novel integrative rehabilitation system for the upper extremity: Design and usability evaluation

PURPOSE

Design and test the usability of a novel virtual rehabilitation system for bimanual training of gravity supported arms, pronation/supination, grasp strengthening, and finger extension.

METHODS

A robotic rehabilitation table, therapeutic game controllers, and adaptive rehabilitation games were developed. The rehabilitation table lifted/lowered and tilted up/down to modulate gravity loading. Arms movement was measured simultaneously, allowing bilateral training. Therapeutic games adapted through a baseline process. Four healthy adults performed four usability evaluation sessions each, and provided feedback using the USE questionnaire and custom questions. Participant's game play performance was sampled and analyzed, and system modifications made between sessions.

RESULTS

Participants played four sessions of about 50 minutes each, with training difficulty gradually increasing. Participants averaged a total of 6,300 arm repetitions, 2,200 grasp counts, and 2,100 finger extensions when adding counts for each upper extremity. USE questionnaire data averaged 5.1/7 rating, indicative of usefulness, ease of use, ease of learning, and satisfaction with the system. Subjective feedback on the custom evaluation form was 84% favorable.

CONCLUSIONS

The novel system was well-accepted, induced high repetition counts, and the usability study helped optimize it and achieve satisfaction. Future studies include examining effectiveness of the novel system when training patients acute post-stroke.

Measuring and Improving User Experience Through Artificial Intelligence-Aided Design

This paper aims to propose a methodology for measuring user experience (UX) by using artificial intelligence-aided design (AIAD) technology in mobile application design. Unlike the traditional assistance design tools, AIAD focuses on the rational use of artificial intelligence (AI) technology to measure and improve UX since conventional data collection methods (such as user interview and user observation) for user behavior data are inefficient and time-consuming. We propose to obtain user behavior data from logs of mobile application. In order to protect the privacy of users, only a few dimensions of information is used in the process of browsing and operating mobile application. The goal of the proposed methodology is to make the deep neural network model simulate the user's experience in the process of operating a mobile application as much as possible. We design and use projected pages of application to train neural networks for specific tasks. These projected pages consist of the click information of all users in the process of completing a certain task. Thus, features of user behavior can be aggregated and mapped in the connection layers and the hidden layers. Finally, the optimized design is executed on the social communication application to verify the efficiency of the proposed methodology.

Compliance with design principles: a case study of a widely used laboratory information system

Background

Laboratory information systems are widely used health information systems that have the potential to improve health care quality. Despite their benefits, many studies have indicated problems with user interaction with these systems due to poor interface design.

Aims

To evaluate the usability of a laboratory information system.

Methods

In this descriptive, cross-sectional study, we used heuristic evaluation to examine the user interface design of a laboratory information system in an academic hospital affiliated with Kerman University of Medical Sciences in 2017. This system is also used in 59 other Iranian hospitals .We investigated the usability of different parts of the usability of a laboratory information system (outpatient admission, inpatient admission, sample collection, and test result reporting). Data were collected using a standard form based on the heuristic evaluation method, and categorized based on their severity and violated heuristics. The content validity was confirmed by 3 medical informatics specialists.

Results

We identified 162 usability problems. In terms of the heuristics, the highest number of problems concerned flexibility and efficiency of use (n = 32, 19.75%) and the lowest concerned help users recognize, diagnose, and recover from errors (n = 2, 1.23%). In terms of different modules of the system, the highest number of problems (n = 51, 31.48%) concerned outpatient admission and the lowest (n = 29, 17.9%) concerned sample collection. In terms of severity, 45.06% of the problems were rated as major.

Conclusions

Despite widespread use of laboratory information systems, their user interface design has usability problems that diminish the quality of user interaction with these systems and may affect the quality of health care. Consideration of standards and principles for user interface design, such as the heuristics used in this study, could improve system usability.

Design and Usability Evaluation of Mobile Voice-Added Food Reporting for Elderly People: Randomized Controlled Trial

BACKGROUND

Advances in voice technology have raised new possibilities for apps related to daily health maintenance. However, the usability of such technologies for older users remains unclear and requires further investigation.

OBJECTIVE

We designed and evaluated two innovative mobile voice-added apps for food intake reporting, namely voice-only reporting (VOR) and voice-button reporting (VBR). Each app features a unique interactive procedure for reporting food intake. With VOR, users verbally report the main contents of each dish, while VBR provides both voice and existing touch screen inputs for food intake reporting. The relative usability of the two apps was assessed through the metrics of accuracy, efficiency, and user perception.

METHODS

The two mobile apps were compared in a head-to-head parallel randomized trial evaluation. A group of 57 adults aged 60-90 years (12 male and 45 female participants) was recruited from a retirement community and randomized into two experimental groups, that is, VOR (n=30) and VBR (n=27) groups. Both groups were tested using the same set of 17 food items including dishes and beverages selected and allocated to present distinct breakfast, lunch, and dinner meals. All participants used a 7-inch tablet computer for the test. The resulting data were analyzed to evaluate reporting accuracy and time efficiency, and the system usability scale (SUS) was used to measure user perception.

RESULTS

For eight error types identified in the experiment, the VBR group participants were significantly (P<.001) more error prone owing to the required use of button-tapping actions. The highest error rates in the VOR group were related to incomprehensible reporting speech (28/420, 6.7%), while the highest error rates in the VBR group were related to failure to make required button taps (39/378, 10.3%). The VOR group required significantly (P<.001) less time to complete food reporting. The overall subjective reactions of the two groups based on the SUS surpassed the benchmark and were not significantly different (P=.20).

CONCLUSIONS

Experimental results showed that VOR outperformed VBR, suggesting that voice-only food input reporting is preferable for elderly users. Voice-added apps offer a potential mechanism for the self-management of dietary intake by elderly users. Our study contributes an evidence-based evaluation of prototype design and selection under a user-centered design model. The results provide a useful reference for selecting optimal user interaction design.

TRIAL REGISTRATION

International Standard Randomized Controlled Trial Registry ISRCTN17335889; http://www.isrctn.com/ISRCTN17335889.

Usability Evaluations of Mobile Mental Health Technologies: Systematic Review

Background

Many mobile health (mHealth) apps for mental health have been made available in recent years. Although there is reason to be optimistic about their effect on improving health and increasing access to care, there is a call for more knowledge concerning how mHealth apps are used in practice.

Objective

This study aimed to review the literature on how usability is being addressed and measured in mHealth interventions for mental health problems.

Methods

We conducted a systematic literature review through a search for peer-reviewed studies published between 2001 and 2018 in the following electronic databases: EMBASE, CINAHL, PsycINFO, PubMed, and Web of Science. Two reviewers independently assessed all abstracts against the inclusion and exclusion criteria, following the Preferred Reporting Items for Systematic Review and Meta-Analysis guidelines.

Results

A total of 299 studies were initially identified based on the inclusion keywords. Following a review of the title, abstract, and full text, 42 studies were found that fulfilled the criteria, most of which evaluated usability with patients (n=29) and health care providers (n=11) as opposed to healthy users (n=8) and were directed at a wide variety of mental health problems (n=24). Half of the studies set out to evaluate usability (n=21), and the remainder focused on feasibility (n=10) or acceptability (n=10). Regarding the maturity of the evaluated systems, most were either prototypes or previously tested versions of the technology, and the studies included few accounts of sketching and participatory design processes. The most common reason referred to for developing mobile mental health apps was the availability of mobile devices to users, their popularity, and how people in general became accustomed to using them for various purposes.

Conclusions

This study provides a detailed account of how evidence of usability of mHealth apps is gathered in the form of usability evaluations from the perspective of computer science and human-computer interaction, including how users feature in the evaluation, how the study objectives and outcomes are stated, which research methods and techniques are used, and what the notion of mobility features is for mHealth apps. Most studies described their methods as trials, gathered data from a small sample size, and carried out a summative evaluation using a single questionnaire, which indicates that usability evaluation was not the main focus. As many studies described using an adapted version of a standard usability questionnaire, there may be a need for developing a standardized mHealth usability questionnaire.

Usability evaluation of an eHealth intervention for family carers of individuals affected by psychosis: A mixed-method study

Background

Existing research suggests that eHealth interventions targeting family carers of individuals with long-term illness offer a promising approach to care delivery. In particular, digital psychoeducational interventions with interactive psychosocial support are well-received with high rates of satisfaction and acceptability. However, development of such interventions for psychosis carers is lacking. We developed a multi-component eHealth intervention specifically for carers of individuals affected by psychosis, called COPe-support (Carers fOr People with Psychosis e-support).

Objective

Using mixed methods to evaluate usability, system heuristics and perceived acceptability, we conducted a usability study to establish the suitability of the intervention prototype for the target user group.

Methods

Twenty-three carers were recruited to the study and participated in a think-aloud test or a remote online trial of the intervention. Qualitative feedback, post-use System Usability Scale (SUS) scores, and real-world usage data collected from the tests were analysed. These were also supplemented with heuristic evaluation data provided by an independent eLearning technology expert.

Results

Participants evaluated the intervention content as useful and helpful, and indicated that the system had satisfactory usability with a mean SUS score of 73%, above the usability quality benchmark threshold. Study results identified some minor usability issues, which were corroborated with the eLearning expert’s heuristic evaluation findings. We used these results to refine the COPe-support intervention.

Conclusions

The usability study with end-users and service providers identified real-life usage and usability issues. The study results helped us refine COPe-support and its delivery strategy before its launch as part of a large-scale clinical trial.

Assistive game controller for artificial intelligence-enhanced telerehabilitation post-stroke

Off-the-shelf gaming technology is designed for young, fit, and motor intact individuals. Artificial intelligence (AI) has a role in making controllers and therapeutic games adaptable to the disabled. Post-stroke rehabilitation outcomes can be enhanced by gaming technology within the home to enable engaging telerehabilitation. BrightBrainer™ Grasp (BBG) is a novel therapeutic game controller designed to adapt to arm and hand impairments post-stroke. It mediates intensive arm reach, grasp and finger extension training and has the ability to track relevant outcomes. The newly designed controller uses BrightBrainer gamification system with AI technology to provide automatic adaptation, requiring minimal clinician input. This article describes the BBG design, hardware, force and movement detection and calibration, and its integration with the therapeutic games. The use of AI in adapting a library of custom therapeutic games is also described. Results of a usability study with healthy individuals and related design modifications are presented, with implications for future trials.

Usability evaluation of communication service robot for the elderly

BACKGROUND

The Republic of Korea has become the most rapidly aging society and is anticipated to enter the post-aged society in 2026. As the population ages, demand for senior-friendly products has gradually increased. Under the demands for these circumstances, usability evaluation aimed to make these products more competitive in terms of quality has been perceived as a critical means. In particular, ergonomically wrong-designed products could prove fatal to elderly people.

OBJECTIVE

The purpose of the study is to develop qualitative usability evaluation criteria of communication service robot, one of the senior-friendly products, for elderly people.

METHODS

To develop the usability testing criteria of a communication service robot for elderly people. The following was carried out: product selection, selection of target product, development of leading indicators, correction by experts, and preliminary evaluation according to the scenario and development of core indicators. For this, a draft questionnaire was developed for the elderly at around age 60. After small group tests and interviews, the experts modified the initial draft to the usability evaluation criteria of communication service robot for elderly people.

RESULTS

Development indicators include four subscales: safety, controllability, efficiency, and satisfaction. All the subscales passed the reliability criteria by four groups of elderly people, divided by gender and the familiarity witth smart move devices.

CONCLUSIONS

Development indicators cover a wider area of user experiences of the communication service robot and are a good measurement tool to help both the users and developers of the service robot for elderly people.

High-Fidelity Prototyping for Mobile Electronic Data Collection Forms Through Design and User Evaluation

Background

Mobile data collection systems are often difficult to use for nontechnical or novice users. This can be attributed to the fact that developers of such tools do not adequately involve end users in the design and development of product features and functions, which often creates interaction challenges.

Objective

The main objective of this study was to assess the guidelines for form design using high-fidelity prototypes developed based on end-user preferences. We also sought to investigate the association between the results from the System Usability Scale (SUS) and those from the Study Tailored Evaluation Questionnaire (STEQ) after the evaluation. In addition, we sought to recommend some practical guidelines for the implementation of the group testing approach particularly in low-resource settings during mobile form design.

Methods

We developed a Web-based high-fidelity prototype using Axure RP 8. A total of 30 research assistants (RAs) evaluated this prototype in March 2018 by completing the given tasks during 1 common session. An STEQ comprising 13 affirmative statements and the commonly used and validated SUS were administered to evaluate the usability and user experience after interaction with the prototype. The STEQ evaluation was summarized using frequencies in an Excel sheet while the SUS scores were calculated based on whether the statement was positive (user selection minus 1) or negative (5 minus user selection). These were summed up and the score contributions multiplied by 2.5 to give the overall form usability from each participant.

Results

Of the RAs, 80% (24/30) appreciated the form progress indication, found the form navigation easy, and were satisfied with the error messages. The results gave a SUS average score of 70.4 (SD 11.7), which is above the recommended average SUS score of 68, meaning that the usability of the prototype was above average. The scores from the STEQ, on the other hand, indicated a 70% (21/30) level of agreement with the affirmative evaluation statements. The results from the 2 instruments indicated a fair level of user satisfaction and a strong positive association as shown by the Pearson correlation value of .623 (P<.01).

Conclusions

A high-fidelity prototype was used to give the users experience with a product they would likely use in their work. Group testing was done because of scarcity of resources such as costs and time involved especially in low-income countries. If embraced, this approach could help assess user needs of the diverse user groups. With proper preparation and the right infrastructure at an affordable cost, usability testing could lead to the development of highly usable forms. The study thus makes recommendations on the practical guidelines for the implementation of the group testing approach particularly in low-resource settings during mobile form design.

A Human⁻Machine Interface Based on Eye Tracking for Controlling and Monitoring a Smart Home Using the Internet of Things

People with severe disabilities may have difficulties when interacting with their home devices due to the limitations inherent to their disability. Simple home activities may even be impossible for this group of people. Although much work has been devoted to proposing new assistive technologies to improve the lives of people with disabilities, some studies have found that the abandonment of such technologies is quite high. This work presents a new assistive system based on eye tracking for controlling and monitoring a smart home, based on the Internet of Things, which was developed following concepts of user-centered design and usability. With this system, a person with severe disabilities was able to control everyday equipment in her residence, such as lamps, television, fan, and radio. In addition, her caregiver was able to monitor remotely, by Internet, her use of the system in real time. Additionally, the user interface developed here has some functionalities that allowed improving the usability of the system as a whole. The experiments were divided into two steps. In the first step, the assistive system was assembled in an actual home where tests were conducted with 29 participants without disabilities. In the second step, the system was tested with online monitoring for seven days by a person with severe disability (end-user), in her own home, not only to increase convenience and comfort, but also so that the system could be tested where it would in fact be used. At the end of both steps, all the participants answered the System Usability Scale (SUS) questionnaire, which showed that both the group of participants without disabilities and the person with severe disabilities evaluated the assistive system with mean scores of 89.9 and 92.5, respectively.

Evaluation of a Patient-Facing Digital Prototype for Perioperative Risk Assessment

Preparations for anesthesiological management of patients build on preoperative patient self-reports concerning risk factors and comorbidities. In this setting, electronic documentation could facilitate innovative computerized functions, although patient-facing digital questionnaires require appropriate tools that patients can access effectively. To explore the feasibility of an electronic application for preoperative data acquisition directly from patients, a digital, tablet-based prototypical application has been developed within a user-centered design process in order to replace a previously used paper-based anamnesis sheet for perioperative risk evaluation. The implemented prototype has been extensively tested and iteratively improved to progressively provide an easy-to-use data entry function. To assess the suitability of this tool for everyday data acquisition by patients and physicians and to identify usability problems, the stepwise development process was accompanied by a heuristic evaluation as well as a think-aloud evaluation, while another 56 participating patients completed a feedback sheet according to ISO 9241/10. The latter method detected additional usability problems that occurred during the use of the application, which contributed to iterative improvements of the prototype. Throughout the development process, 81 issues were identified and largely resolved. After these revisions of the prototype, the number of problems found per tester decreased from 4.75 to 0.96, while the overall rating increased to 6.14 out of 7 points (SD = 1.2). These improvements demonstrate the value and efficiency of such a user-centered design process and illustrate that a user-friendly patient-facing digital data entry can replace preoperative paper questionnaires for anesthesiological management.

Usability Evaluation of a Dashboard for Home Care Nurses

The introduction of electronic health records has produced many challenges for clinicians. These include integrating technology into clinical workflow and fragmentation of relevant information across systems. Dashboards, which use visualized data to summarize key patient information, have the potential to address these issues. In this article, we outline a usability evaluation of a dashboard designed for home care nurses. An iterative design process was used, which consisted of (1) contextual inquiry (observation and interviews) with two home care nurses; (2) rapid feedback on paper prototypes of the dashboard (10 nurses); and (3) usability evaluation of the final dashboard prototype (20 nurses). Usability methods and assessments included observation of nurses interacting with the dashboard, the system usability scale, and the Questionnaire for User Interaction Satisfaction short form. The dashboard prototype was deemed to have high usability (mean system usability scale, 73.2 [SD, 18.8]) and was positively evaluated by nurse users. It is important to ensure that technology solutions such as the one proposed in this article are designed with clinical users in mind, to meet their information needs. The design elements of the dashboard outlined in this article could be translated to other electronic health records used in home care settings.

Teleconsultation Using Mobile Phones for Diagnosis and Acute Care of Burn Injuries Among Emergency Physicians: Mixed-Methods Study

Background

The referral process in acute care remains challenging in many areas including burn care. Mobile phone apps designed explicitly for medical referrals and consultations could streamline the referral process by using structured templates and integrating features specific to different specialties. However, as these apps are competing with commercial chat services, usability becomes a crucial factor for successful uptake.

Objective

The aim of this study was to assess the usability of a mobile phone app for remote consultations and referrals of burn injuries.

Methods

A total of 24 emergency doctors and 4 burns consultants were recruited for the study. A mixed-methods approach was used including a usability questionnaire and a think-aloud interview. Think-aloud sessions were video-recorded, and content analysis was undertaken with predefined codes relating to the following 3 themes: ease of use, usefulness of content, and technology-induced errors.

Results

The users perceived the app to be easy to use and useful, but some problems were identified. Issues relating to usability were associated with navigation, such as scrolling and zooming. Users also had problems in understanding the meaning of some icons and terminologies. Sometimes, some users felt limited by predefined options, and they wanted to be able to freely express their clinical findings.

Conclusions

We found that users faced problems mainly with navigation when the app did not work in the same way as the other apps that were frequently used. Our study also resonates with previous findings that when using standardized templates, the systems should also allow the user to express their clinical findings in their own words.

A Mobile Health Intervention for HIV Prevention Among Racially and Ethnically Diverse Young Men: Usability Evaluation

BACKGROUND

Mobile health (mHealth) apps have the potential to be a useful mode of delivering HIV prevention information, particularly for young men (13-24 years) who account for 21% of new HIV diagnoses in the United States. We translated an existing evidence-based, face-to-face HIV prevention curriculum into a portable platform and developed a mobile Web app: MyPEEPS Mobile.

OBJECTIVE

The purpose of this study was to assess the usability of MyPEEPS Mobile from both expert and end user perspectives.

METHODS

We conducted a heuristic evaluation with five experts in informatics to identify violations of usability principles and end user usability testing with 20 young men aged 15 to 18 years in New York, NY, Birmingham, AL, and Chicago, IL to identify potential obstacles to their use of the app.

RESULTS

Mean scores of the overall severity of the identified heuristic violations rated by experts ranged from 0.4 and 2.6 (0=no usability problem to 4=usability catastrophe). Overall, our end users successfully completed the tasks associated with use case scenarios and provided comments/recommendations on improving usability of MyPEEPS Mobile. The mean of the overall Post-Study System Usability Questionnaire scores rated by the end users was 1.63 (SD 0.65), reflecting strong user acceptance of the app.

CONCLUSIONS

The comments made by experts and end users will be used to refine MyPEEPS Mobile prior to a pilot study assessing the acceptability of the app across diverse sexual minority young men in their everyday lives.

A Web-Based Treatment Decision Support Tool for Patients With Advanced Knee Arthritis: Evaluation of User Interface and Content Design

Background

Data-driven surgical decisions will ensure proper use and timing of surgical care. We developed a Web-based patient-centered treatment decision and assessment tool to guide treatment decisions among patients with advanced knee osteoarthritis who are considering total knee replacement surgery.

Objective

The aim of this study was to examine user experience and acceptance of the Web-based treatment decision support tool among older adults.

Methods

User-centered formative and summative evaluations were conducted for the tool. A sample of 28 patients who were considering total knee replacement participated in the study. Participants’ responses to the user interface design, the clarity of information, as well as usefulness, satisfaction, and acceptance of the tool were collected through qualitative (ie, individual patient interviews) and quantitative (ie, standardized Computer System Usability Questionnaire) methods.

Results

Participants were older adults with a mean age of 63 (SD 11) years. Three-quarters of them had no technical questions using the tool. User interface design recommendations included larger fonts, bigger buttons, less colors, simpler navigation without extra “next page” click, less mouse movement, and clearer illustrations with simple graphs. Color-coded bar charts and outcome-specific graphs with positive action were easiest for them to understand the outcomes data. Questionnaire data revealed high satisfaction with the tool usefulness and interface quality, and also showed ease of use of the tool, regardless of age or educational status.

Conclusions

We evaluated the usability of a patient-centered decision support tool designed for advanced knee arthritis patients to facilitate their knee osteoarthritis treatment decision making. The lessons learned can inform other decision support tools to improve interface and content design for older patients’ use.

Human-centered design of a personal health record system for metabolic syndrome management based on the ISO 9241-210:2010 standard

Background

Previous studies have demonstrated the effectiveness of information and communication technologies to support healthy lifestyle interventions. In particular, personal health record systems (PHR-Ss) empower self-care, essential to support lifestyle changes. Approaches such as the user-centered design (UCD), which is already a standard within the software industry (ISO 9241-210:2010), provide specifications and guidelines to guarantee user acceptance and quality of eHealth systems. However, no single PHR-S for metabolic syndrome (MS) developed following the recommendations of the ISO 9241-210:2010 specification has been found in the literature.

Objective

The aim of this study was to describe the development of a PHR-S for the management of MS according to the principles and recommendations of the ISO 9241-210 standard.

Methods

The proposed PHR-S was developed using a formal software development process which, in addition to the traditional activities of any software process, included the principles and recommendations of the ISO 9241-210 standard. To gather user information, a survey sample of 1,187 individuals, eight interviews, and a focus group with seven people were performed. Throughout five iterations, three prototypes were built. Potential users of each system evaluated each prototype. The quality attributes of efficiency, effectiveness, and user satisfaction were assessed using metrics defined in the ISO/IEC 25022 standard.

Results

The following results were obtained: 1) a technology profile from 1,187 individuals at risk for MS from the city of Popayan, Colombia, identifying that 75.2% of the people use the Internet and 51% had a smartphone; 2) a PHR-S to manage MS developed (the PHR-S has the following five main functionalities: record the five MS risk factors, share these measures with health care professionals, and three educational modules on nutrition, stress management, and a physical activity); and 3) usability tests on each prototype obtaining the following results: 100% effectiveness, 100% efficiency, and 84.2 points in the system usability scale.

Conclusion

The software development methodology used was based on the ISO 9241-210 standard, which allowed the development team to maintain a focus on user’s needs and requirements throughout the project, which resulted in an increased satisfaction and acceptance of the system. Additionally, the establishment of a multidisciplinary team allowed the application of considerations not only from the disciplines of software engineering and health sciences but also from other disciplines such as graphical design and media communication. Finally, usability testing allowed the observation of flaws in the designs, which helped to improve the solution.

AIM - Agile Instrumented Monitoring for Improving User Experience of Participation in HealthIT Development

Participation of healthcare professionals in information technology development has emerged as an important challenge. As end-users, the professionals are willing to participate in the development activities, but their experiences on the current methods of participation remain mostly negative. There is lack of applicable methods for meeting the needs of agile development approach and scaling up to the largest implementation projects, while maintaining the interest of the professional users to participate in development activities and keeping up their ability to continue working in a productive manner. In this paper, we describe the Agile Instrumented Monitoring as a methodology, based on the methods of instrumented usability evaluation, for improving user experience in HealthIT development. The contribution of the proposed methodology is analyzed in relation to activities of whole iteration cycle and chosen usability evaluation methods, while the user experience of participation is addressed regarding healthcare professionals. Prospective weak and strong market tests for AIM are discussed in the conclusions for future work.

A Facebook-Based Obesity Prevention Program for Korean American Adolescents: Usability Evaluation

INTRODUCTION

Adolescent obesity is one of the most serious global public health challenges. Social networking sites are currently popular among adolescents. Therefore, the obesity prevention program for Korean American adolescents was developed on the most popular social networking site, Facebook. The purpose of this study was to evaluate the usability of a culturally tailored Facebook-based obesity prevention program for Korean American adolescents (Healthy Teens).

METHOD

An explorative descriptive design of usability testing was used. Usability testing employing one-on-one observation, the think-aloud method, audio taping, screen activity capture, and surveys was performed. Twenty participants were recruited from two Korean language schools (mean age, 15.40 ± 1.50 years). Recruitment and user testing was performed between February and April 2014. Content analysis, using the inductive coding approach, was performed by three coders to analyze transcriptions. Descriptive statistics were used to analyze quantitative data including demographic characteristics, perceived usability, eHealth literacy, and health behaviors.

RESULTS

Testing revealed several usability issues in content, appearance, and navigation. Participants' comments regarding content were positive. Although the Facebook platform provided limited flexibility with respect to building the site, participants described the program's appearance as appropriate. Most participants did not experience difficulty in navigating the program.

DISCUSSION

Our preliminary findings indicated that participants perceived the Healthy Teens program as usable and useful. This program could be used as a robust platform for the delivery of health education to adolescents. Further research is required to assess the effects of Facebook-based programs on adolescent obesity prevention.

Usability Evaluation Methods for Gesture-Based Games: A Systematic Review

Background

Gestural interaction systems are increasingly being used, mainly in games, expanding the idea of entertainment and providing experiences with the purpose of promoting better physical and/or mental health. Therefore, it is necessary to establish mechanisms for evaluating the usability of these interfaces, which make gestures the basis of interaction, to achieve a balance between functionality and ease of use.

Objective

This study aims to present the results of a systematic review focused on usability evaluation methods for gesture-based games, considering devices with motion-sensing capability. We considered the usability methods used, the common interface issues, and the strategies adopted to build good gesture-based games.

Methods

The research was centered on four electronic databases: IEEE, Association for Computing Machinery (ACM), Springer, and Science Direct from September 4 to 21, 2015. Within 1427 studies evaluated, 10 matched the eligibility criteria. As a requirement, we considered studies about gesture-based games, Kinect and/or Wii as devices, and the use of a usability method to evaluate the user interface.

Results

In the 10 studies found, there was no standardization in the methods because they considered diverse analysis variables. Heterogeneously, authors used different instruments to evaluate gesture-based interfaces and no default approach was proposed. Questionnaires were the most used instruments (70%, 7/10), followed by interviews (30%, 3/10), and observation and video recording (20%, 2/10). Moreover, 60% (6/10) of the studies used gesture-based serious games to evaluate the performance of elderly participants in rehabilitation tasks. This highlights the need for creating an evaluation protocol for older adults to provide a user-friendly interface according to the user’s age and limitations.

Conclusions

Through this study, we conclude this field is in need of a usability evaluation method for serious games, especially games for older adults, and that the definition of a methodology and a test protocol may offer the user more comfort, welfare, and confidence.

Development of a Teen-Friendly Health Education Program on Facebook: Lessons Learned

INTRODUCTION

Facebook is the most popular online platform among adolescents and can be an effective medium to deliver health education. Although Korean American (KA) adolescents are at risk of obesity, a culturally tailored health education program is not available for them. Thus, our research team developed a health education program for KA adolescents on Facebook called "Healthy Teens." The aim of this study was to discuss important lessons learned through the program development process.

METHOD

This program includes culturally tailored learning modules about healthy eating and physical activity. The program was developed on the basis of the social cognitive theory, and the online program was developed by applying Web usability principles for adolescents. Upon completion, the usability of the program was assessed using heuristic evaluation.

RESULTS

The findings from the heuristic evaluation showed that the Healthy Teens program was usable for KA adolescents.

DISCUSSION

The findings from this study will assist researchers who are planning to build similar Facebook-based health education programs.

Feasibility and usability of a wearable orthotic for stroke survivors with hand impairment

PURPOSE

The concept of a vibrating wristband, to improve dextrous hand function of stroke survivors, was recently proposed with clinical results and is referred to as 'TheraBracelet' in this paper. The purpose of this study was to demonstrate feasibility of a portable, wearable TheraBracelet, and to apply usability evaluation techniques to assess potential demands of TheraBracelet and to identify critical improvement needs of the prototype.

METHOD

A prototype was developed with a vibrating element housed in an elastic wristband and connected to a wearable electronics box via a cable. Expectation for TheraBracelet and evaluation of the prototype were obtained from 10 chronic stroke survivors using surveys before and after using the prototype and House of Quality analysis.

RESULTS

The survey for expectation showed stroke survivors' willingness to try out TheraBracelet at a low cost. The survey evaluating the prototype showed that the current prototype was overall satisfactory with a mean rating of 3.7 out of 5. The House of Quality analysis revealed that the priority improvement needs for the prototype are to improve clinical knowledge on long-term effectiveness, reduce cost, ease donning/doffing and waterproof.

CONCLUSIONS

This study presents a potential for a low-cost wearable hand orthotic likable by stroke survivors. Implications for Rehabilitation Feasibility for a portable wearable wristband-type hand orthotic was demonstrated. The survey showed stroke survivors are willing to try such an orthotic at low cost. The current prototype was rated overall satisfactory by stroke survivors. This study provides a potential for a low-cost wearable hand orthotic likable by stroke survivors.

Semantic Indexing of Medical Learning Objects: Medical Students' Usage of a Semantic Network

BACKGROUND

The Semantically Annotated Media (SAM) project aims to provide a flexible platform for searching, browsing, and indexing medical learning objects (MLOs) based on a semantic network derived from established classification systems. Primarily, SAM supports the Aachen emedia skills lab, but SAM is ready for indexing distributed content and the Simple Knowledge Organizing System standard provides a means for easily upgrading or even exchanging SAM's semantic network. There is a lack of research addressing the usability of MLO indexes or search portals like SAM and the user behavior with such platforms.

OBJECTIVE

The purpose of this study was to assess the usability of SAM by investigating characteristic user behavior of medical students accessing MLOs via SAM.

METHODS

In this study, we chose a mixed-methods approach. Lean usability testing was combined with usability inspection by having the participants complete four typical usage scenarios before filling out a questionnaire. The questionnaire was based on the IsoMetrics usability inventory. Direct user interaction with SAM (mouse clicks and pages accessed) was logged.

RESULTS

The study analyzed the typical usage patterns and habits of students using a semantic network for accessing MLOs. Four scenarios capturing characteristics of typical tasks to be solved by using SAM yielded high ratings of usability items and showed good results concerning the consistency of indexing by different users. Long-tail phenomena emerge as they are typical for a collaborative Web 2.0 platform. Suitable but nonetheless rarely used keywords were assigned to MLOs by some users.

CONCLUSIONS

It is possible to develop a Web-based tool with high usability and acceptance for indexing and retrieval of MLOs. SAM can be applied to indexing multicentered repositories of MLOs collaboratively.

Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics

OBJECTIVE

Mobile health (mHealth) systems are becoming more common for chronic disease management, but usability studies are still needed on patients' perspectives and mHealth interaction performance. This deficiency is addressed by our quantitative usability study of a mHealth diabetes system evaluating patients' task performance, satisfaction, and the relationship of these measures to user characteristics.

MATERIALS AND METHODS

We used metrics in the International Organization for Standardization (ISO) 9241-11 standard. After standardized training, 10 patients performed representative tasks and were assessed on individual task success, errors, efficiency (time on task), satisfaction (System Usability Scale [SUS]) and user characteristics.

RESULTS

Tasks of exporting and correcting values proved the most difficult, had the most errors, the lowest task success rates, and consumed the longest times on task. The average SUS satisfaction score was 80.5, indicating good but not excellent system usability. Data trends showed males were more successful in task completion, and younger participants had higher performance scores. Educational level did not influence performance, but a more recent diabetes diagnosis did. Patients with more experience in information technology (IT) also had higher performance rates.

DISCUSSION

Difficult task performance indicated areas for redesign. Our methods can assist others in identifying areas in need of improvement. Data about user background and IT skills also showed how user characteristics influence performance and can provide future considerations for targeted mHealth designs.

CONCLUSION

Using the ISO 9241-11 usability standard, the SUS instrument for satisfaction and measuring user characteristics provided objective measures of patients' experienced usability. These could serve as an exemplar for standardized, quantitative methods for usability studies on mHealth systems.

Using Eye Trackers for Usability Evaluation of Health Information Technology: A Systematic Literature Review

Background

Eye-tracking technology has been used to measure human cognitive processes and has the potential to improve the usability of health information technology (HIT). However, it is still unclear how the eye-tracking method can be integrated with other traditional usability methodologies to achieve its full potential.

Objective

The objective of this study was to report on HIT evaluation studies that have used eye-tracker technology, and to envision the potential use of eye-tracking technology in future research.

Methods

We used four reference databases to initially identify 5248 related papers, which resulted in only 9 articles that met our inclusion criteria.

Results

Eye-tracking technology was useful in finding usability problems in many ways, but is still in its infancy for HIT usability evaluation. Limited types of HITs have been evaluated by eye trackers, and there has been a lack of evaluation research in natural settings.

Conclusions

More research should be done in natural settings to discover the real contextual-based usability problems of clinical and mobile HITs using eye-tracking technology with more standardized methodologies and guidance.

Functional analysis of interfaces in U.S. military electronic health record system using UFuRT framework

The overall aim of this study is to evaluate the usability of U.S. military electronic health record (EHR) system AHLTA using a systematic work-centered evaluation framework UFuRT --- User, Functional, Representational, and Task Analysis. This paper with the focus of Functional Analysis (FA) of AHLTA explores operationalizable methods to study functions supported by user interfaces. A system hierarchy was created to map and uniquely identify all items on the interfaces. These items were then classified independently by 2 evaluators as Operations or Objects. Operations were further classified as either Domain or Overhead function. With acceptable inter-rater agreement, of the 1996 items in the interfaces, 61% were operations, around one fourth of which were Overhead functions. Overhead functions are hypothesized to be targets to be redesigned for improvements in usability.

Usability of an adaptive computer assistant that improves self-care and health literacy of older adults

OBJECTIVES

We developed an adaptive computer assistant for the supervision of diabetics' self-care, to support limiting illness and need for acute treatment, and improve health literacy. This assistant monitors self-care activities logged in the patient's electronic diary. Accordingly, it provides context-aware feedback. The objective was to evaluate whether older adults in general can make use of the computer assistant and to compare an adaptive computer assistant with a fixed one, concerning its usability and contribution to health literacy.

METHODS

We conducted a laboratory experiment in the Georgia Tech Aware Home wherein 28 older adults participated in a usability evaluation of the computer assistant, while engaged in scenarios reflecting normal and health-critical situations. We evaluated the assistant on effectiveness, efficiency, satisfaction, and educational value. Finally, we studied the moderating effects of the subjects' personal characteristics.

RESULTS

Logging self-care tasks and receiving feedback from the computer assistant enhanced the subjects' knowledge of diabetes. The adaptive assistant was more effective in dealing with normal and health-critical situations, and, generally, it led to more time efficiency. Subjects' personal characteristics had substantial effects on the effectiveness and efficiency of the two computer assistants.

CONCLUSIONS

Older adults were able to use the adaptive computer assistant. In addition, it had a positive effect on the development of health literacy. The assistant has the potential to support older diabetics' self care while maintaining quality of life.

Usability Evaluation of the Spatial OLAP Visualization and Analysis Tool (SOVAT)

Increasingly sophisticated technologies, such as On-Line Analytical Processing (OLAP) and Geospatial Information Systems (GIS), are being leveraged for conducting community health assessments (CHA). Little is known about the usability of OLAP and GIS interfaces with respect to CHA. We conducted an iterative usability evaluation of the Spatial OLAP Visualization and Analysis Tool (SOVAT), a software application that combines OLAP and GIS. A total of nine graduate students and six community health researchers were asked to think-aloud while completing five CHA questions using SOVAT. The sessions were analyzed after every three participants and changes to the interface were made based on the findings. Measures included elapsed time, answers provided, erroneous actions, and satisfaction. Traditional OLAP interface features were poorly understood by participants and combined OLAP-GIS features needed to be better emphasized. The results suggest that the changes made to the SOVAT interface resulted in increases in both usability and user satisfaction.