The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool

Evaluating the Usability and Quality of a Clinical Mobile App for Assisting Physicians in Head Computed Tomography Scan Ordering: Mixed Methods Study

BACKGROUND

Among the numerous factors contributing to health care providers' engagement with mobile apps, including user characteristics (eg, dexterity, anatomy, and attitude) and mobile features (eg, screen and button size), usability and quality of apps have been introduced as the most influential factors.

OBJECTIVE

This study aims to investigate the usability and quality of the Head Computed Tomography Scan Appropriateness Criteria (HAC) mobile app for physicians' computed tomography scan ordering.

METHODS

Our study design was primarily based on methodological triangulation by using mixed methods research involving quantitative and qualitative think-aloud usability testing, quantitative analysis of the Mobile Apps Rating Scale (MARS) for quality assessment, and debriefing across 3 phases. In total, 16 medical interns participated in quality assessment and testing usability characteristics, including efficiency, effectiveness, learnability, errors, and satisfaction with the HAC app.

RESULTS

The efficiency and effectiveness of the HAC app were deemed satisfactory, with ratings of 97.8% and 96.9%, respectively. MARS assessment scale indicated the overall favorable quality score of the HAC app (82 out of 100). Scoring 4 MARS subscales, Information (73.37 out of 100) and Engagement (73.48 out of 100) had the lowest scores, while Aesthetics had the highest score (87.86 out of 100). Analysis of the items in each MARS subscale revealed that in the Engagement subscale, the lowest score of the HAC app was "customization" (63.6 out of 100). In the Functionality subscale, the HAC app's lowest value was "performance" (67.4 out of 100). Qualitative think-aloud usability testing of the HAC app found notable usability issues grouped into 8 main categories: lack of finger-friendly touch targets, poor search capabilities, input problems, inefficient data presentation and information control, unclear control and confirmation, lack of predictive capabilities, poor assistance and support, and unclear navigation logic.

CONCLUSIONS

Evaluating the quality and usability of mobile apps using a mixed methods approach provides valuable information about their functionality and disadvantages. It is highly recommended to embrace a more holistic and mixed methods strategy when evaluating mobile apps, because results from a single method imperfectly reflect trustworthy and reliable information regarding the usability and quality of apps.

Development of minimum data set and dashboard for monitoring adverse events in radiology departments

Background

To reduce the risk of errors, patient safety monitoring in the medical imaging department is crucial. Interventions are required and these can be provided as a framework for documenting, reporting, evaluating, and recognizing events that pose a threat to patient safety. The aim of this study was to develop minimum data set and dashboard for monitoring adverse events in radiology departments.

Material and methods

This developmental research was conducted in multiple phases, including content determination using the Delphi technique; database designing using SQL Server; user interface (UI) building using PHP; and dashboard evaluation in three aspects: the accuracy of calculating; UI requirements; and usability.

Results

This study identified 26 patient safety (PS) performance metrics and 110 PS-related significant data components organized into 14 major groupings as the system contents. The UI was built with three tabs: pre-procedure, intra-procedure, and post-procedure. The evaluation results proved the technical feasibility of the dashboard. Finally, the dashboard's usability was highly rated (76.3 out of 100).

Conclusion

The dashboard can be used to supplement datasets to obtain a more accurate picture of the PS condition and to draw attention to characteristics that professionals might otherwise overlook or undervalue.

Usability evaluation of emergency information systems in educational hospitals in Kerman, Iran

BACKGROUND

Smart and practical health information systems and applications with fewer errors are crucial for healthcare facilities. One method that ensures the proper design of health information systems (HIS) and applications is usability evaluation.

OBJECTIVE

This study aimed to evaluate the usability of the emergency information systems used at the emergency departments of four educational hospitals in Kerman, Iran.

METHOD

This study was conducted in two phases. In the first phase, the information systems' errors and shortages were identified using a semi-structured questionnaire by users (nurses and the IT staff). In the second phase, based on the results of the first phase, two questionnaires were designed for each group of users to their opinions about the usability of the emergency information systems.

RESULTS

The average score of "reducing and facilitating user's daily activities" was significantly different among hospitals (p = 0.03). Shahid Beheshti Hospital obtained the lowest usability score (17.5), and Afzalipour Hospital received the highest usability score (21.75). Moreover, the average score in "use of the HIS" for nurses and IT staff was 2.93 and 3.54 on a scale of 5, respectively.

CONCLUSION

Usability evaluation of health information systems is essential to ensure that these systems provide sufficient and accurate information and requirements for users and health care providers. Also, modifying health information systems based on the user views and expectations improves the quality of the system and user-system Interaction.

Developing and testing the usability, acceptability, and future implementation of the Whole Day Matters Tool and User Guide for primary care providers using think-aloud, near-live, and interview procedures

BACKGROUND

Canada's 24-Hour Movement Guidelines for Adults have shifted the focus from considering movement behaviours (i.e., physical activity, sedentary behaviour, and sleep) separately to a 24-h paradigm, which considers how they are integrated. Accordingly, primary care providers (PCPs) have the opportunity to improve their practice to promote all movement behaviours cohesively. However, PCPs have faced barriers to discussing physical activity alone (e.g., time, competing priorities, inadequate training), leading to low frequency of physical activity discussions. Consequently, discussing three movement behaviours may seem challenging. Tools to facilitate primary care discussions about physical activity have been developed and used; however, few have undergone usability testing and none have integrated all movement behaviours. Following a synthesis of physical activity, sedentary behaviour, and sleep tools for PCPs, we developed the Whole Day Matters Tool and User Guide that incorporate all movement behaviours. The present study aimed to explore PCPs' perceptions on the usability, acceptability, and future implementation of the Whole Day Matters Tool and User Guide to improve their relevancy among PCPs.

METHODS

Twenty-six PCPs were observed and audio-video recorded while using the Tool and User Guide in a think-aloud procedure, then in a near-live encounter with a mock service-user. A debriefing interview using a guide informed by Normalization Process Theory followed. Recordings were transcribed verbatim and analysed using content analysis and a critical friend to enhance rigour.

RESULTS

PCPs valued aspects of the Tool and User Guide including their structure, user-friendliness, visual appeal, and multi-behaviour focus and suggested modifications to improve usability and acceptability. Findings are further discussed in the context of Normalization Process Theory and previous literature.

CONCLUSIONS

The Tool and User Guide were revised, including adding plain language, reordering and renaming sections, reducing text, and clarifying instructions. Results also informed the addition of a Preamble and a Handout for adults accessing care (i.e., patients/clients/service-users) to explain the evidence underpinning the 24-Hour Movement Guidelines for Adults and support a person-centered approach. These four resources (i.e., Tool, User Guide, Preamble, Handout) have since undergone a consensus building process to arrive at their final versions before being disseminated into primary care practice.

Moderated digital social therapy for young people with emerging mental health problems: A user-centered mixed-method design and usability study

Introduction

Over 25% of Dutch young people are psychologically unhealthy. Individual and societal consequences that follow from having mental health complaints at this age are substantial. Young people need care which is often unavailable. ENgage YOung people earlY (ENYOY) is a moderated digital social therapy-platform that aims to help youngsters with emerging mental health complaints. Comprehensive research is being conducted into the effects and to optimize and implement the ENYOY-platform throughout the Netherlands. The aim of this study is to explore the usability and user experience of the ENYOY-platform.

Methods

A user-centered mixed-method design was chosen. 26 young people aged 16-25 with emerging mental health complaints participated. Semi-structured interviews were conducted to explore usability, user-friendliness, impact, accessibility, inclusivity, and connection (Phase 1). Phase 2 assessed usability problems using the concurrent and retrospective Think Aloud-method. User experience and perceived helpfulness were assessed using a 10-point rating scale and semi-structured interviews (Phase 3). The Health Information Technology Usability Evaluation Scale (Health-ITUES; Phase 1) and System Usability Scale (SUS; Phase 2 and 3) were administered. Qualitative data was analyzed using thematic analysis. Task completion rate and time were tracked and usability problems were categorized using the Nielsen's rating scale (Phase 2).

Results

Adequate to high usability was found (Phase 1 Health-ITUES 4.0(0.34); Phase 2 SUS 69,5(13,70); Phase 3 SUS 71,6(5,63)). Findings from Phase 1 (N = 10) indicated that users viewed ENYOY as a user-friendly, safe, accessible, and inclusive initiative which helped them reduce their mental health complaints and improve quality of life. Phase 2 (N = 10) uncovered 18 usability problems of which 5 of major severity (e.g. troubles accessing the platform). Findings from Phase 3 (N = 6) suggested that users perceived the coaching calls the most helpful [9(0.71)] followed by the therapy content [6.25(1.41)]. Users liked the social networking aspect but rated it least helpful [6(2.1)] due to inactivity.

Conclusion

The ENYOY-platform has been found to have adequate to high usability and positive user experiences were reported. All findings will be transferred to the developmental team to improve the platform. Other evaluation methods and paring these with quantitative outcomes could provide additional insight in future research.

Expert-centric design inspection of branded apps: a close look at marketing and interface design features

Purpose

Although brands have developed mobile applications (apps) to offer consumers new experiences, low app usage numbers indicate the need to develop a systematic, practical evaluation framework for branded app design that specifies concrete design features.

Design/methodology/approach

An expert review provides an overview of the design of current branded apps. On the basis of an extensive literature review, this article classifies state-of-the-art design features for branded apps according to a proposed evaluation framework that includes human–computer interaction (HCI)–related and marketing-related evaluation criteria. In an application of these evaluation criteria, the authors evaluate 73 branded apps issued by 11 top fast-moving consumer goods (FMCG) brands.

Findings

The expert review identifies strengths and weaknesses that are common to the design of current branded apps. These findings inform the set of design recommendations that this article offers, which includes 14 features common to all types of apps and 9 features specific to particular types of apps.

Practical implications

This research offers practical implications for app designers, who need to address design dimensions contained in the proposed framework including the HCI-related (mobile, social and user experience design features) and marketing-related (branding and customer relationship management design features) to create effective branded apps.

Originality/value

Design elements identified in prior literature remain abstract and do not prescribe a systematic or pragmatic approach to using them in practice. This study takes a multidisciplinary perspective (HCI, marketing and design science) to establish a practical evaluation framework for branded app designs.

Evaluation of an Electronic Medical Record Module for Nursing Documentation in Paediatric Palliative Care: Involvement of Nurses with a Think-Aloud Approach

BACKGROUND

Paediatric palliative care (PPC) is a noncurative approach to the care of children and adolescents with life-limiting and life-threatening illnesses. Electronic medical records (EMRs) play an important role in documenting such complex processes. Despite their benefits, they can introduce unintended consequences if future users are not involved in their development.

AIM

The aim of this study was to evaluate the acceptance of a novel module for nursing documentation by nurses working in the context of PPC.

METHODS

An observational study employing concurrent think-aloud and semi-structured qualitative interviews were conducted with 11 nurses working in PPC. Based on the main determinants of the unified theory of acceptance and use of technology (UTAUT), data were analysed using qualitative content analysis.

RESULTS

The main determinants of UTAUT were found to potentially influence acceptance of the novel module. Participants perceived the module to be self-explanatory and intuitive. Some adaptations, such as the reduction of fragmentation in the display, the optimization of confusing mouseover fields, and the use of familiar nursing terminology, are reasonable ways of increasing software adoption.

CONCLUSIONS

After adaptation of the modules based on the results, further evaluation with the participation of future users is required.

COLAEVA: Visual Analytics and Data Mining Web-Based Tool for Virtual Coaching of Older Adult Populations

The global population is aging in an unprecedented manner and the challenges for improving the lives of older adults are currently both a strong priority in the political and healthcare arena. In this sense, preventive measures and telemedicine have the potential to play an important role in improving the number of healthy years older adults may experience and virtual coaching is a promising research area to support this process. This paper presents COLAEVA, an interactive web application for older adult population clustering and evolution analysis. Its objective is to support caregivers in the design, validation and refinement of coaching plans adapted to specific population groups. COLAEVA enables coaching caregivers to interactively group similar older adults based on preliminary assessment data, using AI features, and to evaluate the influence of coaching plans once the final assessment is carried out for a baseline comparison. To evaluate COLAEVA, a usability test was carried out with 9 test participants obtaining an average SUS score of 71.1. Moreover, COLAEVA is available online to use and explore.

A Mobile Health App for Premarital Related Word Advice (AmeeSehat): Usability and Usefulness Evaluation Study

Background:Premarital counseling demonstrates promising approach for enhancing interaction between couples and to prevent deterioration in the well-being of relationships. Mobile health (mHealth) apps may provide an appropriate solution to this issue, as many potential brides and grooms are frequent users of internet or web-based health info. Aim:This research aimed to measure the usability and usefulness of the AmeeSehat App as interpreted by potential end users. Method: This study employed a mixed method carried out in one of the public-health centers in East Jakarta, Indonesia with the future bride and groom. Think Aloud (TA) sessions with participants (n=10 final users and 3 experts) and quantitative assessment with 60 participants were held.A think aloud meetings have been captured, transcripts, and analyzed. The usability issues were evaluated using Nielsen severity scale and the Device Usability Scale (SUS). The usefulness of the AmeeSehat App was assessed using a tool to evaluate an improvement in healthy family knowledge, attitude, and intention, and readiness to marry. Results: The SUS score obtained (mean=78.4, SD=6.83), and none of the participants experienced (severe) problems with usability. Only 1 usability problems have been reported with a severity of 3 (major), namely terminology issues. The change after using AmeeSehat was statistically significant different as McNemar test revealed p <0.05. Conclusion:This researcher revealed that, in terms of its usability and usefulness, the AmeeSehat app is feasible to future brides and grooms. Participants considered AmeeSehat beneficial because they had the opportunity to learn more about a stable home life in preparation for marriage. Further assessment should be carried out in the form of a randomized controlled trial to test AmeeSehat's effectiveness in enhancing awareness, attitude, intention and readiness.

Differences in echocardiography interpretation techniques among trainees and expert readers

Background

Trainees learn transthoracic echocardiogram (TTE) interpretation through independently completing and reviewing selected portions of the study with experts. The diagnostic accuracy of novice TTE interpretation is known to be low and schema for reading TTEs systematically are lacking. The purpose of our study is to identify techniques experts use while reading TTEs which could be used to more effectively teach novice readers.

Methods

We performed a prospective qualitative case study to observe how experts and trainees interpret TTEs in an academic institution using a concurrent think aloud (CTA) method. Three TTEs of intermediate complexity were given to 3 advanced imaging fellows, 3 first year fellows and 3 expert TTE readers Participants filled out a report while reading and described aloud their thought processes. Sessions were video and audiotaped for analysis.

Results

Experts and advanced fellows used specific techniques that novices did not including: previewing studies, reviewing multiple images simultaneously, having flexibility in image review order and disease coding, and saving hardest elements to code for the end. Direct observation of TTE reading informed trainee inefficiencies and was a well-received educational tool.

Conclusions

In this single centered study we identified several unique approaches experts use to interpret TTEs which may be teachable to novices. Although limited in generalizability the findings of this study suggests that a more systematic approach to TTE interpretation, using techniques found in experts, might be of significant value for trainees. Further study is needed to evaluate teaching practices at other institutions and to assess whether implementation of these techniques by novices improves can improve their diagnostic accuracy and efficiency of reading at an earlier stage in their training.

Improving the Management of Atrial Fibrillation in General Practice: Protocol for a Mixed Methods Study

BACKGROUND

Atrial fibrillation (AF) is one of the commonest arrhythmias observed in general practice. The thromboembolic complications of AF include transient ischemic attack, stroke, and pulmonary embolism. Early recognition of AF can lead to early intervention with managing the risks of these complications.

OBJECTIVE

The primary aim of this study is to investigate if patients are managed in general practice according to current national guidelines. In addition, the study will evaluate the impact of direct oral anticoagulant use with respect to AF complications in a real-world dataset. The secondary aims of the study are to develop a dashboard that will allow monitoring the management of AF in general practice and evaluate the usability of the dashboard.

METHODS

The study was conducted in 2 phases. The initial phase was a quantitative analysis of routinely collected primary care data from the Oxford Royal College of General Practitioners Research and Surveillance Center (RCGP RSC) sentinel network database. AF cases from 2009 to 2019 were identified. The study investigated the impact of the use of anticoagulants on complications of AF over this time period. We used this dataset to examine how AF was managed in primary care during the last decade. The second phase involved development of an online dashboard for monitoring management of AF in general practice. We conducted a usability evaluation for the dashboard to identify usability issues and performed enhancements to improve usability.

RESULTS

We received funding for both phases in January 2019 and received approval from the RCGP RSC research committee in March 2019. We completed data extraction for phase 1 in May 2019 and completed analysis in December 2019. We completed building the AF dashboard in May 2019. We started recruiting participants for phase 1 in May 2019 and concluded data collection in July 2019. We completed data analysis for phase 2 in October 2019. The results are expected to be published in the second half of 2020. As of October 2020, the publications reporting the results are under review.

CONCLUSIONS

Results of this study will provide an insight into the current trends in management of AF using real-world data from the Oxford RCGP RSC database. We anticipate that the outcomes of this study will be used to guide the development and implementation of an audit-based intervention tool to assist practitioners in identifying and managing AF in primary care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR1-10.2196/21259.

Atrial fibrillation dashboard evaluation using the think aloud protocol

BACKGROUND

Atrial fibrillation (AF) is a common cardiac arrhythmia which is a major risk factor for stroke, transient ischaemic attacks and increased mortality. Primary care management of AF can significantly reduce these risks. We carried out an evaluation to asses the usability of an AF dashboard developed to improve data quality and the quality of care.

METHOD

We developed an online dashboard about the quality of AF management for general practices of the Oxford Royal College of General Practitioners Research and Surveillance Centre network. The dashboard displays (1) case ascertainment, (2) a calculation of stroke and haemorrhage risk to assess whether the benefits of anticogulants outweigh their risk, (3) prescriptions of different types of anticoagulant and (4) if prescribed anticoagulant is at the correct dose. We conducted the think aloud evaluation, involving 24 dashboard users to improve its usability.

RESULTS

Analysis of 24 transcripts received produced 120 individual feedback items (ie, verbalised tasks) that were mapped across five usability problem classes. We enhanced the dashboard based on evaluation feedback to encourage adoption by general practices participating in the sentinel network.

CONCLUSIONS

The think aloud evaluation provided useful insights into important usability issues that require further development. Our enhanced AF dashboard was acceptable to clinicians and its impact on data quality and care should be assessed in a formal study.

How to interact with medical terminologies? Formative usability evaluations comparing three approaches for supporting the use of MedDRA by pharmacovigilance specialists

BACKGROUND

Medical terminologies are commonly used in medicine. For instance, to answer a pharmacovigilance question, pharmacovigilance specialists (PVS) search in a pharmacovigilance database for reports in relation to a given drug. To do that, they first need to identify all MedDRA terms that might have been used to code an adverse reaction in the database, but terms may be numerous and difficult to select as they may belong to different parts of the hierarchy. In previous studies, three tools have been developed to help PVS identify and group all relevant MedDRA terms using three different approaches: forms, structured query-builder, and icons. Yet, a poor usability of the tools may increase PVS' workload and reduce their performance. This study aims to evaluate, compare and improve the three tools during two rounds of formative usability evaluation.

METHODS

First, a cognitive walkthrough was performed. Based on the design recommendations obtained from this evaluation, designers made modifications to their tools to improve usability. Once this re-engineering phase completed, six PVS took part in a usability test: difficulties, errors and verbalizations during their interaction with the three tools were collected. Their satisfaction was measured through the System Usability Scale. The design recommendations issued from the tests were used to adapt the tools.

RESULTS

All tools had usability problems related to the lack of guidance in the graphical user interface (e.g., unintuitive labels). In two tools, the use of the SNOMED CT to find MedDRA terms hampered their use because French PVS were not used to it. For the most obvious and common terms, the icons-based interface would appear to be more useful. For the less frequently used MedDRA terms or those distributed in different parts of the hierarchy, the structured query-builder would be preferable thanks to its great power and flexibility. The form-based tool seems to be a compromise.

CONCLUSION

These evaluations made it possible to identify the strengths of each tool but also their weaknesses to address them before further evaluation. Next step is to assess the acceptability of tools and the expressiveness of their results to help identify and group MedDRA terms.

Using Intervention Mapping to Developmentally Adapt an Online HIV Risk Reduction Program for Adolescent Men Who Have Sex with Men

Adolescent men who have sex with men (AMSM) experience a dramatic health disparity in HIV, accounting for over 80% of new diagnoses among youth. Current evidence-based HIV prevention programs, however, focus primarily on adults and heterosexual youth, thereby missing the unique experiences and socio-environmental contexts of AMSM aged 13-18. To address these gaps, we used the Intervention Mapping (IM) protocol to developmentally adapt an existing evidence-based online HIV risk reduction program (i.e., Keep it Up!/KIU!), originally designed for young adult MSM aged 18-29, into a new intervention called SMART Squad. Using a hybrid of IM creation and adaptation tasks, we specified three behavioral outcomes and identified corresponding performance objectives for SMART Squad based on the original goals of KIU!. We constructed matrices of change objectives using determinants from the Information-Motivation-Behavioral Skills model, modifying them for the younger population with additional theoretical and empirical evidence and expert review. SMART Squad activities were operationalized from theory-based behavior change methods matched to the change objectives and guided by program themes, components, and scope imported from KIU!. The final SMART Squad intervention comprises 6 episodes/modules delivered in 2 sessions plus 2 booster episodes occurring 1 and 3 months after the main program. It is currently being evaluated nationally as part of a stepped-care package of 3 programs, in which the receipt and sequencing of interventions is tailored to individual AMSM development and needs. Despite substantial changes to KIU!, IM was a useful method for retaining the hypothesized essential elements of the eHealth HIV risk reduction program. Challenges and recommendations for future researchers and practitioners are discussed.

OBINTER: A Holistic Approach to Catalyse the Self-Management of Chronic Obesity

Obesity is a preventable chronic condition that, in 2016, affected more than 1.9 billion people globally. Several factors have been identified that have a positive impact on long-term weight loss programs such as personalized recommendations, adherence strategies, weight and diet follow-up or physical activity tracking. Recently, various applications have been developed which help patients to self-manage their condition. These apps implement either one or some of these identified factors; however, there is not a single application that combines all of them following a holistic approach. In this context, we developed the OBINTER platform, which assists patients during the weight loss process by targeting user engagement during the longer term. The solution includes a mobile application which allows users to fill out dietetic questionnaires, receive dietetic and nutraceutical plans, track the evolution of their weight and adherence to the diet, as well as track their physical activity via a wearable device. Furthermore, an adherence strategy has been developed as a tool to foster the app usage during the whole weight loss process. In this paper, we present how the OBINTER approach gathers all of these features as well as the positive results of a usability testing study performed to assess the performance and usability of the OBINTER platform.

Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment

Simulations and games bring the possibility to research complex processes of managerial decision-making. However, this modern field requires adequate methodological procedures. Many authors recommend the use of a combination of concurrent think-aloud (CTA) or retrospective think-aloud (RTA) with eye-tracking to investigate cognitive processes such as decision-making. Nevertheless, previous studies have little or no consideration of the possible differential impact of both think-aloud methods on data provided by eye-tracking. Therefore, the main aim of this study is to compare and assess if and how these methods differ in terms of their impact on eye-tracking. The experiment was conducted for this purpose. Participants were 14 managers who played a specific simulation game with CTA use and 17 managers who played the same game with RTA use. The results empirically prove that CTA significantly distorts data provided by eye-tracking, whereas data gathered when RTA is used, provide independent pieces of evidence about the participants' behavior. These findings suggest that RTA is more suitable for combined use with eye-tracking for the purpose of the research of decision-making in the game environment.

Usability and Usefulness of a Mobile Health App for Pregnancy-Related Work Advice: Mixed-Methods Approach

Background

Pregnant women are often unaware of the potential risks that working conditions can cause to them and their unborn child. A mobile health (mHealth) app, the Pregnancy and Work (P and W) app, developed by a multidisciplinary team and based on an evidence-based guideline for occupational physicians, aims to provide advice on work adjustment during pregnancy.

Objective

This study evaluates the usability of the mHealth P and W app and the perceived usefulness of the work advice, the main goal of the app, by potential end users.

Methods

A total of 12 working pregnant women participated in think aloud usability sessions and performed 9 tasks. All think aloud sessions were recorded, transcribed, and coanalyzed. The usability problems were rated for their severity in accordance with Nielsen severity scale. The completion rates and time taken for completion of tasks were registered. In addition, participants were questioned on demographics and user characteristics and were asked to evaluate the value of the app by filling in the Intrinsic Motivation Inventory (IMI) score and the System Usability Scale (SUS) questionnaire.

Results

In total, 82 usability problems with a severity ≥1 were identified, of which 40 had severity ≥3. The main usability problems concerned the interpretation of terminology used in the app’s questionnaires and difficulties in finding and understanding the work advice. Furthermore, 10 out of 12 participants were able to open the work advice page in the app. Only 7 out of these 10 participants understood and intended to follow the work advice. The overall mean IMI score was relatively high (5 out of 7), indicating that the participants did indeed value the use of the app. This IMI score corresponded to the overall mean SUS score (68 out of 100) and the mean grade given to the P and W app (7 out of 10).

Conclusions

This think aloud usability study showed that the information provided in the P and W app was considered valuable by the end users, working pregnant women, and it meets their needs; however, usability issues severely impacted the perceived usefulness of the work advice given in the app.

Building Usability Knowledge for Health Information Technology: A Usability-Oriented Analysis of Incident Reports

BACKGROUND

The contribution of usability flaws to patient safety issues is acknowledged but not well-investigated. Free-text descriptions of incident reports may provide useful data to identify the connection between health information technology (HIT) usability flaws and patient safety.

OBJECTIVES

This article examines the feasibility of using incident reports about HIT to learn about the usability flaws that affect patient safety. We posed three questions: (1) To what extent can we gain knowledge about usability issues from incident reports? (2) What types of usability flaws, related usage problems, and negative outcomes are reported in incidents reports? (3) What are the reported usability issues that give rise to patient safety issues?

METHODS

A sample of 359 reports from the U.S. Food and Drug Administration Manufacturer and User Facility Device Experience database was examined. Descriptions of usability flaws, usage problems, and negative outcomes were extracted and categorized. A supplementary analysis was performed on the incidents which contained the full chain going from a usability flaw up to a patient safety issue to identify the usability issues that gave rise to patient safety incidents.

RESULTS

A total of 249 reports were included. We found that incident reports can provide knowledge about usability flaws, usage problems, and negative outcomes. Thirty-six incidents report how usability flaws affected patient safety (ranging from incidents without consequence, to death) involving electronic patient scales, imaging systems, and HIT for medication management. The most significant class of involved usability flaws is related to the reliability, the understandability, and the availability of the clinical information.

CONCLUSION

Incidents reports involving HIT are an exploitable source of information to learn about usability flaws and their effects on patient safety. Results can be used to convince all stakeholders involved in the HIT system lifecycle that usability should be considered seriously to prevent patient safety incidents.

Timeline representation of clinical data: usability and added value for pharmacovigilance

Background

Pharmacovigilance consists in monitoring and preventing the occurrence of adverse drug reactions (ADR). This activity requires the collection and analysis of data from the patient record or any other sources to find clues of a causality link between the drug and the ADR. This can be time-consuming because often patient data are heterogeneous and scattered in several files. To facilitate this task, we developed a timeline prototype to gather and classify patient data according to their chronology. Here, we evaluated its usability and quantified its contribution to routine pharmacovigilance using real ADR cases.

Methods

The timeline prototype was assessed using the biomedical data warehouse eHOP (from entrepôt de données biomédicales de l’HOPital) of the Rennes University Hospital Centre. First, the prototype usability was tested by six experts of the Regional Pharmacovigilance Centre of Rennes. Their experience was assessed with the MORAE software and a System and Usability Scale (SUS) questionnaire. Then, to quantify the timeline contribution to pharmacovigilance routine practice, three of them were asked to investigate possible ADR cases with the “Usual method” (analysis of electronic health record data with the DxCare software) or the “Timeline method”. The time to complete the task and the data quality in their reports (using the vigiGrade Completeness score) were recorded and compared between methods.

Results

All participants completed their tasks. The usability could be considered almost excellent with an average SUS score of 82.5/100. The time to complete the assessment was comparable between methods (P = 0.38) as well as the average vigiGrade Completeness of the data collected with the two methods (P = 0.49).

Conclusions

The results showed a good general level of usability for the timeline prototype. Conversely, no difference in terms of the time spent on each ADR case and data quality was found compared with the usual method. However, this absence of difference between the timeline and the usual tools that have been in use for several years suggests a potential use in pharmacovigilance especially because the testers asked to continue using the timeline after the evaluation.

Electronic supplementary material

The online version of this article (10.1186/s12911-018-0667-x) contains supplementary material, which is available to authorized users.

Usability Evaluation of Three Admission and Medical Records Subsystems Integrated into Nationwide Hospital Information Systems: Heuristic Evaluation

Introduction:

Usability is one of the quality criteria for information systems and its weakness is one of the main barriers to the adoption of these systems. The purpose of this study was to evaluate the usability of admission and medical records module of three widely used hospital information systems (HISs).

Methods:

In this descriptive study the usability of admission and medical records module of three HISs (HIS1, HIS2, and HIS3) was evaluated using heuristic evaluation method. For each HIS, three expert users of the same system assessed the user interface independently, completed a usability evaluation checklist, and rated severity of each identified problem. The checklist was based on Nielsen’s heuristics. For each HIS, three heuristics that have the highest and lowest problem rates and greatest severity of problems were categorized into three separate groups. The results were analyzed using descriptive statistics.

Results:

Although HIS1 and HIS2 were used in more hospitals than HIS3, the results showed that the usability problem rates of them were significantly higher than HIS3. The heuristics of “help and documentation”, “flexibility and efficiency of use”, and “visibility of system status” in the three HISs were categorized into the “highest rate of problems”, “lowest rate of problems”, and “highest severity of problems” groups, respectively. The heuristics of “diagnose and recover from errors”, “error prevention”, and “help and documentation” in HIS1 and HIS2 were categorized into the “highest rate of problems” group.

Conclusions:

The results of this study and previous studies show that the most common usability problems with HISs are related to heuristics of “help and documentation”, “error prevention”, and “help users recognize, diagnose and recover from errors.” Also, the large number of hospitals using one HIS does not demonstrate its high usability to others.

Concept and implementation of a study dashboard module for a continuous monitoring of trial recruitment and documentation

BACKGROUND

The difficulty of managing patient recruitment and documentation for clinical trials prompts a demand for instruments for closely monitoring these critical but unpredictable processes. Increasingly adopted Electronic Data Capture (EDC) applications provide novel opportunities to reutilize stored information for an efficient management of traceable trial workflows. In related clinical and administrative settings, so-called digital dashboards that continuously visualize time-dependent parameters have recently met a growing acceptance. To investigate the technical feasibility of a study dashboard for monitoring the progress of patient recruitment and trial documentation, we set out to develop a propositional prototype in the form of a separate software module.

METHODS

After narrowing down functional requirements in semi-structured interviews with study coordinators, we analyzed available interfaces of a locally deployed EDC application, and designed the prototypical study dashboard based on previous findings. The module thereby leveraged a standardized export format in order to extract and import relevant trial data into a clinical data warehouse. Web-based reporting tools then facilitated the definition of diverse views, including diagrams of the progress of patient accrual and form completion at different granularity levels. To estimate the utility of the dashboard and its compatibility with current workflows, we interviewed study coordinators after a demonstration of sample outputs from ongoing trials.

RESULTS

The employed tools promoted a rapid development. Displays of the implemented dashboard are organized around an entry page that integrates key metrics for available studies, and which links to more detailed information such as study-specific enrollment per center. The interviewed experts commented that the included graphical summaries appeared suitable for detecting that something was generally amiss, although practical remedies would mostly depend on additional information such as access to the original patient-specific data. The dependency on a separate application was seen as a downside. Interestingly, the prospective users warned that in some situations knowledge of specific accrual statistics might undermine blinding in a subtle yet intricate fashion, so ignorance of certain patient features was seen as sometimes preferable for reproducibility.

DISCUSSION

Our proposed study dashboard graphically recaps key progress indicators of patient accrual and trial documentation. The modular implementation illustrates the technical feasibility of the approach. The use of a study dashboard might introduce certain technical requirements as well as subtle interpretative complexities, which may have to be weighed against potential efficiency gains.

Usability Testing of a National Substance Use Screening Tool Embedded in Electronic Health Records

Background

Screening, brief intervention, and referral to treatment (SBIRT) is currently being implemented into health systems nationally via paper and electronic methods.

Objective

The purpose of this study was to evaluate the integration of an electronic SBIRT tool into an existing paper-based SBIRT clinical workflow in a patient-centered medical home.

Methods

Usability testing was conducted in an academic ambulatory clinic. Two rounds of usability testing were done with medical office assistants (MOAs) using a paper and electronic version of the SBIRT tool, with two and four participants, respectively. Qualitative and quantitative data was analyzed to determine the impact of both tools on clinical workflow. A second round of usability testing was done with the revised electronic version and compared with the first version.

Results

Personal workflow barriers cited in the first round of testing were that the electronic health record (EHR) tool was disruptive to patient’s visits. In Round 2 of testing, MOAs reported favoring the electronic version due to improved layout and the inclusion of an alert system embedded in the EHR. For example, using the system usability scale (SUS), MOAs reported a grade “1” for the statement, “I would like to use this system frequently” during the first round of testing but a “5” during the second round of analysis.

Conclusions

The importance of testing usability of various mediums of tools used in health care screening is highlighted by the findings of this study. In the first round of testing, the electronic tool was reported as less user friendly, being difficult to navigate, and time consuming. Many issues faced in the first generation of the tool were improved in the second generation after usability was evaluated. This study demonstrates how usability testing of an electronic SBRIT tool can help to identify challenges that can impact clinical workflow. However, a limitation of this study was the small sample size of MOAs that participated. The results may have been biased to Northwell Health workers’ perceptions of the SBIRT tool and their specific clinical workflow.