Classification and prioritization of usability problems using an augmented classification scheme

How to co-design a prototype of a clinical practice tool: a framework with practical guidance and a case study

Clinical tools for use in practice-such as medicine reconciliation charts, diagnosis support tools and track-and-trigger charts-are endemic in healthcare, but relatively little attention is given to how to optimise their design. User-centred design approaches and co-design principles offer potential for improving usability and acceptability of clinical tools, but limited practical guidance is currently available. We propose a framework (FRamework for co-dESign of Clinical practice tOols or 'FRESCO') offering practical guidance based on user-centred methods and co-design principles, organised in five steps: (1) establish a multidisciplinary advisory group; (2) develop initial drafts of the prototype; (3) conduct think-aloud usability evaluations; (4) test in clinical simulations; (5) generate a final prototype informed by workshops. We applied the framework in a case study to support co-design of a prototype track-and-trigger chart for detecting and responding to possible fetal deterioration during labour. This started with establishing an advisory group of 22 members with varied expertise. Two initial draft prototypes were developed-one based on a version produced by national bodies, and the other with similar content but designed using human factors principles. Think-aloud usability evaluations of these prototypes were conducted with 15 professionals, and the findings used to inform co-design of an improved draft prototype. This was tested with 52 maternity professionals from five maternity units through clinical simulations. Analysis of these simulations and six workshops were used to co-design the final prototype to the point of readiness for large-scale testing. By codifying existing methods and principles into a single framework, FRESCO supported mobilisation of the expertise and ingenuity of diverse stakeholders to co-design a prototype track-and-trigger chart in an area of pressing service need. Subject to further evaluation, the framework has potential for application beyond the area of clinical practice in which it was applied.

Evaluating the usability of a cancer registry system using Cognitive Walkthrough, and assessing user agreement with its problems

OBJECTIVE/AIM

Good design of cancer registry systems makes them easy to use, while poor design of their user interfaces leads to user dissatisfaction and resistance. The objective of this study was to evaluate the usability of a cancer registry system using Cognitive Walkthrough (CW) and to assess users' agreement with its usability problems.

METHODS

CW was used to evaluate the registry system. We developed a checklist to help evaluators speed up the evaluation process, a problems form to collect the usability issues identified by the evaluators, and a problems severity form to determine the severity of problems by the evaluators. The problems were classified into two categories according to the CW questions and the system tasks. The agreement of the users with the system problems was examined by an online questionnaire. Users' agreement with the problems was then analyzed using the Interclass Correlation Coefficient in the SPSS 22 (Statistical Package for Social Science).

RESULTS

In this study, 114 problems were identified. In the categorization of problems based on the CW questions, 41% (n = 47) of the problems concerned the issue of "users do not know what to do at each stage of working with the system", 24% (n = 27) were classified as "users cannot link what they intend to do with system controls", and 22% (n = 25) were related to "user's lack of understanding of the system processes". Based on user tasks, about 36% (n = 41) of the problems were related to "removing patient duplication" and 33% (n = 38) were related to "registration of patient identification information". User agreement with the problems was high (CI 95% = 0.9 (0.96, 0.98)).

CONCLUSION

System problems often originate from user ignorance about what to do at each stage of using the system. Also, half of the system problems concern a mismatch between what users want to do and the system controls, or a lack of understanding about what the system does at different stages. Therefore, to avoid user confusion, designers should use clues and guides on the screen for users, design controls consistent with the user model of thinking, and provide appropriate feedback after each user action to help users understand what the system is doing. The high agreement of users with the problems showed that in the absence of users system designers can use CW to identify the problems that users face in the real environment.

Usability evaluation of mHealth apps for elderly individuals: a scoping review

BACKGROUND

Usability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.

METHODS

Electronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.

RESULTS

Ninety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.

CONCLUSION

In the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Human-Centered Design of Mobile Health Apps for Older Adults: Systematic Review and Narrative Synthesis

BACKGROUND

The world is aging. The number of older patients is on the rise, and along with it comes the burden of noncommunicable diseases, both clinical and economic. Attempts with mobile health (mHealth) have been made to remedy the situation with promising outcomes. Researchers have adopted human-centered design (HCD) in mHealth creation to ensure those promises become a reality.

OBJECTIVE

This systematic review aims to explore existing literature on relevant primary research and case studies to (1) illustrate how HCD can be used to create mHealth solutions for older adults and (2) summarize the overall process with recommendations specific to the older population.

METHODS

We conducted a systematic review to address the study objectives. IEEE Xplore, Medline via Ovid, PubMed, and Scopus were searched for HCD research of mHealth solutions for older adults. Two independent reviewers then included the papers if they (1) were written in English, (2) included participants equal to or older than 60 years old, (3) were primary research, and (4) reported about mHealth apps and their HCD developments from start to finish. The 2 reviewers continued to assess the included studies' qualities using the Mixed Methods Appraisal Tool (MMAT). A narrative synthesis was then carried out and completed.

RESULTS

Eight studies passed the eligibility criteria: 5 were mixed methods studies and 3 were case studies. Some studies were about the same mHealth projects with a total of 5 mHealth apps. The included studies differed in HCD goals, target groups, and details of their HCD methodologies. The HCD process was explored through narrative synthesis in 4 steps according to the International Standardization Organization (ISO) standard 9241-210: (1) understand and specify the context of use, (2) specify the user requirements, (3) produce design solutions to meet these requirements, and (4) evaluate the designs against requirements. The overall process and recommendations unique to older adults are summarized logically with structural order and time order based on the Minto pyramid principle and ISO 9241-210.

CONCLUSIONS

Findings show that HCD can be used to create mHealth solutions for older adults with positive outcomes. This review has also summarized practical HCD steps and additional suggestions based on existing literature in the subfield. However, evidence-based results are still limited because most included studies lacked details about their sampling methods and did not set objective and quantifiable goals, leading to failure to draw significant conclusions. More studies of HCD application on mHealth for older adults with measurable design goals and rigorous research strategy are warranted.

Conceptualizing Usability for the eHealth Context: Content Analysis of Usability Problems of eHealth Applications

BACKGROUND

Usability tests can be either formative (where the aim is to detect usability problems) or summative (where the aim is to benchmark usability). There are ample formative methods that consider user characteristics and contexts (ie, cognitive walkthroughs, interviews, and verbal protocols). This is especially valuable for eHealth applications, as health conditions can influence user-system interactions. However, most summative usability tests do not consider eHealth-specific factors that could potentially affect the usability of a system. One of the reasons for this is the lack of fine-grained frameworks or models of usability factors that are unique to the eHealth domain.

OBJECTIVE

In this study, we aim to develop an ontology of usability problems, specifically for eHealth applications, with patients as primary end users.

METHODS

We analyzed 8 data sets containing the results of 8 formative usability tests for eHealth applications. These data sets contained 400 usability problems that could be used for analysis. Both inductive and deductive coding were used to create an ontology from 6 data sets, and 2 data sets were used to validate the framework by assessing the intercoder agreement.

RESULTS

We identified 8 main categories of usability factors, including basic system performance, task-technology fit, accessibility, interface design, navigation and structure, information and terminology, guidance and support, and satisfaction. These 8 categories contained a total of 21 factors: 14 general usability factors and 7 eHealth-specific factors. Cohen κ was calculated for 2 data sets on both the category and factor levels, and all Cohen κ values were between 0.62 and 0.67, which is acceptable. Descriptive analysis revealed that approximately 69.5% (278/400) of the usability problems can be considered as general usability factors and 30.5% (122/400) as eHealth-specific usability factors.

CONCLUSIONS

Our ontology provides a detailed overview of the usability factors for eHealth applications. Current usability benchmarking instruments include only a subset of the factors that emerged from our study and are therefore not fully suited for summative evaluations of eHealth applications. Our findings support the development of new usability benchmarking tools for the eHealth domain.

Use of Human-Centered Design to Improve Implementation of Evidence-Based Psychotherapies in Low-Resource Communities: Protocol for Studies Applying a Framework to Assess Usability

BACKGROUND

This paper presents the protocol for the National Institute of Mental Health (NIMH)-funded University of Washington's ALACRITY (Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness) Center (UWAC), which uses human-centered design (HCD) methods to improve the implementation of evidence-based psychosocial interventions (EBPIs). We propose that usability-the degree to which interventions and implementation strategies can be used with ease, efficiency, effectiveness, and satisfaction-is a fundamental, yet poorly understood determinant of implementation.

OBJECTIVE

We present a novel Discover, Design/Build, and Test (DDBT) framework to study usability as an implementation determinant. DDBT will be applied across Center projects to develop scalable and efficient implementation strategies (eg, training tools), modify existing EBPIs to enhance usability, and create usable and nonburdensome decision support tools for quality delivery of EBPIs.

METHODS

Stakeholder participants will be implementation practitioners/intermediaries, mental health clinicians, and patients with mental illness in nonspecialty mental health settings in underresourced communities. Three preplanned projects and 12 pilot studies will employ the DDBT model to (1) identify usability challenges in implementing EBPIs in underresourced settings; (2) iteratively design solutions to overcome these challenges; and (3) compare the solution to the original version of the EPBI or implementation strategy on usability, quality of care, and patient-reported outcomes. The final products from the center will be a streamlined modification and redesign model that will improve the usability of EBPIs and implementation strategies (eg, tools to support EBPI education and decision making); a matrix of modification targets (ie, usability issues) that are both common and unique to EBPIs, strategies, settings, and patient populations; and a compilation of redesign strategies and the relative effectiveness of the redesigned solution compared to the original EBPI or strategy.

RESULTS

The UWAC received institutional review board approval for the three separate studies in March 2018 and was funded in May 2018.

CONCLUSIONS

The outcomes from this center will inform the implementation of EBPIs by identifying cross-cutting features of EBPIs and implementation strategies that influence the use and acceptability of these interventions, actively involving stakeholder clinicians and implementation practitioners in the design of the EBPI modification or implementation strategy solution and identifying the impact of HCD-informed modifications and solutions on intervention effectiveness and quality.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03515226 (https://clinicaltrials.gov/ct2/show/NCT03515226), NCT03514394 (https://clinicaltrials.gov/ct2/show/NCT03514394), and NCT03516513 (https://clinicaltrials.gov/ct2/show/NCT03516513).

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/14990.

A usability study to improve a clinical decision support system for the prescription of antibiotic drugs

OBJECTIVE

A clinical decision support system (CDSS) for empirical antibiotic treatment has the potential to increase appropriate antibiotic use. Before using such a system on a broad scale, it needs to be tailored to the users preferred way of working. We have developed a CDSS for empirical antibiotic treatment in hospitalized adult patients. Here we determined in a usability study if the developed CDSS needed changes.

METHODS

Four prespecified patient cases, based on real life clinical scenarios, were evaluated by 8 medical residents in the study. The "think-aloud" method was used, and sessions were recorded and analyzed afterwards. Usability was assessed by 3 evaluators using an augmented classification scheme, which combines the User Action Framework with severity rating of the usability problems and the assessment of the potential impact of these problems on the final task outcomes.

RESULTS

In total 51 usability problems were identified, which could be grouped into 29 different categories. Most (n = 17/29) of the usability problems were cosmetic problems or minor problems. Eighteen (out of 29) of the usability categories could have an ordering error as a result. Classification of the problems showed that some of the problems would get a low priority based on their severity rating, but got a high priority for their impact on the task outcome. This effectively provided information to prioritize system redesign efforts.

CONCLUSION

Usability studies improve lay-out and functionality of a CDSS for empirical antibiotic treatment, even after development by a multidisciplinary system.

Aging barriers influencing mobile health usability for older adults: A literature based framework (MOLD-US)

BACKGROUND

With the growing population of older adults as a potential user group of mHealth, the need increases for mHealth interventions to address specific aging characteristics of older adults. The existence of aging barriers to computer use is widely acknowledged. Yet, usability studies show that mHealth still fails to be appropriately designed for older adults and their expectations. To enhance designs of mHealth aimed at older adult populations, it is essential to gain insight into aging barriers that impact the usability of mHealth as experienced by these adults.

OBJECTIVES

This study aims to synthesize literature on aging barriers to digital (health) computer use, and explain, map and visualize these barriers in relation to the usability of mHealth by means of a framework.

METHODS

We performed a scoping review to synthesize and summarize reported physical and functional age barriers in relation to digital (mobile) health applications use. Aging barriers reported in the literature were mapped onto usability aspects categorized by Nielsen to explain their influence on user experience of mHealth. A framework (MOLD-US) was developed summarizing the evidence on the influence of aging barriers on mHealth use experienced by older adults.

RESULTS

Four key categories of aging barriers influencing usability of mHealth were identified: cognition, motivation, physical ability and perception. Effective and satisfactory use of mHealth by older adults is complicated by cognition and motivation barriers. Physical ability and perceptual barriers further increase the risk of user errors and fail to notice important interaction tasks. Complexities of medical conditions, such as diminished eye sight related to diabetes or deteriorated motor skills as a result of rheumatism, can cause errors in user interaction.

CONCLUSIONS

This research provides a novel framework for the exploration of aging barriers and their causes influencing mHealth usability in older adults. This framework allows for further systematic empirical testing and analysis of mHealth usability issues, as it enables results to be classified and interpreted based on impediments intrinsic to usability issues experienced by older adults. Importantly, the paper identifies a key need for future research on motivational barriers impeding mhealth use of older adults. More insights are needed in particular to disaggregating normal age related functional changes from specific medical conditions that influence experienced usefulness of mHealth by these adults.

Comparison of two heuristic evaluation methods for evaluating the usability of health information systems

OBJECTIVES

In addition to following the usual Heuristic Evaluation (HE) method, the usability of health information systems can also be evaluated using a checklist. The objective of this study is to compare the performance of these two methods in identifying usability problems of health information systems.

METHODS

Eight evaluators independently evaluated different parts of a Medical Records Information System using two methods of HE (usual and with a checklist). The two methods were compared in terms of the number of problems identified, problem type, and the severity of identified problems.

RESULTS

In all, 192 usability problems were identified by two methods in the Medical Records Information System. This was significantly higher than the number of usability problems identified by the checklist and usual method (148 and 92, respectively) (p < 0.0001). After removing the duplicates, the difference between the number of unique usability problems identified by the checklist method (n = 100) and usual method (n = 44) was significant (p < 0.0001). Differences between the mean severity of the real usability problems (1.83) and those identified by only one of the methods (usual = 2.05, checklist = 1.74) were significant (p = 0.001).

CONCLUSIONS

This study revealed the potential of the two HE methods for identifying usability problems of health information systems. The results demonstrated that the checklist method had significantly better performance in terms of the number of identified usability problems; however, the performance of the usual method for identifying problems of higher severity was significantly better. Although the checklist method can be more efficient for less experienced evaluators, wherever usability is critical, the checklist should be used with caution in usability evaluations.

Human-Centered Design Study: Enhancing the Usability of a Mobile Phone App in an Integrated Falls Risk Detection System for Use by Older Adult Users

BACKGROUND

Design processes such as human-centered design (HCD), which involve the end user throughout the product development and testing process, can be crucial in ensuring that the product meets the needs and capabilities of the user, particularly in terms of safety and user experience. The structured and iterative nature of HCD can often conflict with the necessary rapid product development life-cycles associated with the competitive connected health industry.

OBJECTIVE

The aim of this study was to apply a structured HCD methodology to the development of a smartphone app that was to be used within a connected health fall risk detection system. Our methodology utilizes so called discount usability engineering techniques to minimize the burden on resources during development and maintain a rapid pace of development. This study will provide prospective designers a detailed description of the application of a HCD methodology.

METHODS

A 3-phase methodology was applied. In the first phase, a descriptive "use case" was developed by the system designers and analyzed by both expert stakeholders and end users. The use case described the use of the app and how various actors would interact with it and in what context. A working app prototype and a user manual were then developed based on this feedback and were subjected to a rigorous usability inspection. Further changes were made both to the interface and support documentation. The now advanced prototype was exposed to user testing by end users where further design recommendations were made.

RESULTS

With combined expert and end-user analysis of a comprehensive use case having originally identified 21 problems with the system interface, we have only seen and observed 3 of these problems in user testing, implying that 18 problems were eliminated between phase 1 and 3. Satisfactory ratings were obtained during validation testing by both experts and end users, and final testing by users shows the system requires low mental, physical, and temporal demands according to the NASA Task Load Index (NASA-TLX).

CONCLUSIONS

From our observation of older adults' interactions with smartphone interfaces, there were some recurring themes. Clear and relevant feedback as the user attempts to complete a task is critical. Feedback should include pop-ups, sound tones, color or texture changes, or icon changes to indicate that a function has been completed successfully, such as for the connection sequence. For text feedback, clear and unambiguous language should be used so as not to create anxiety, particularly when it comes to saving data. Warning tones or symbols, such as caution symbols or shrill tones, should only be used if absolutely necessary. Our HCD methodology, designed and implemented based on the principles of the International Standard Organizaton (ISO) 9241-210 standard, produced a functional app interface within a short production cycle, which is now suitable for use by older adults in long term clinical trials.

A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology

Background

Design processes such as human-centered design, which involve the end user throughout the product development and testing process, can be crucial in ensuring that the product meets the needs and capabilities of the user, particularly in terms of safety and user experience. The structured and iterative nature of human-centered design can often present a challenge when design teams are faced with the necessary, rapid, product development life cycles associated with the competitive connected health industry.

Objective

We wanted to derive a structured methodology that followed the principles of human-centered design that would allow designers and developers to ensure that the needs of the user are taken into account throughout the design process, while maintaining a rapid pace of development. In this paper, we present the methodology and its rationale before outlining how it was applied to assess and enhance the usability, human factors, and user experience of a connected health system known as the Wireless Insole for Independent and Safe Elderly Living (WIISEL) system, a system designed to continuously assess fall risk by measuring gait and balance parameters associated with fall risk.

Methods

We derived a three-phase methodology. In Phase 1 we emphasized the construction of a use case document. This document can be used to detail the context of use of the system by utilizing storyboarding, paper prototypes, and mock-ups in conjunction with user interviews to gather insightful user feedback on different proposed concepts. In Phase 2 we emphasized the use of expert usability inspections such as heuristic evaluations and cognitive walkthroughs with small multidisciplinary groups to review the prototypes born out of the Phase 1 feedback. Finally, in Phase 3 we emphasized classical user testing with target end users, using various metrics to measure the user experience and improve the final prototypes.

Results

We report a successful implementation of the methodology for the design and development of a system for detecting and predicting falls in older adults. We describe in detail what testing and evaluation activities we carried out to effectively test the system and overcome usability and human factors problems.

Conclusions

We feel this methodology can be applied to a wide variety of connected health devices and systems. We consider this a methodology that can be scaled to different-sized projects accordingly.

A Multi-Stage Human Factors and Comfort Assessment of Instrumented Insoles Designed for Use in a Connected Health Infrastructure

Wearable electronics are gaining widespread use as enabling technologies, monitoring human physical activity and behavior as part of connected health infrastructures. Attention to human factors and comfort of these devices can greatly positively influence user experience, with a subsequently higher likelihood of user acceptance and lower levels of device rejection. Here, we employ a human factors and comfort assessment methodology grounded in the principles of human-centered design to influence and enhance the design of an instrumented insole. A use case was developed and interrogated by stakeholders, experts, and end users, capturing the context of use and user characteristics for the instrumented insole. This use case informed all stages of the design process through two full design cycles, leading to the development of an initial version 1 and a later version 2 prototype. Each version of the prototype was subjected to an expert human factors inspection and controlled comfort assessment using human volunteers. Structured feedback from the first cycle of testing was the driver of design changes implemented in the version 2 prototype. This prototype was found to have significantly improved human factors and comfort characteristics over the first version of the prototype. Expert inspection found that many of the original problems in the first prototype had been resolved in the second prototype. Furthermore, a comfort assessment of this prototype with a group of young healthy adults showed it to be indistinguishable from their normal footwear. This study demonstrates the power and effectiveness of human factors and comfort assessment methodologies in influencing and improving the design of wearable devices.

[Medication errors related to computerized physician order entry at the hospital: Record and analysis over a period of 4 years]

OBJECTIVES

Computerized physician order entry (CPOE) can generate medication errors. It is necessary to identify them and analyse their causes in order to secure the medication use system.

METHODS

Errors were recorded during the pharmaceutical analysis of prescriptions over a period of 4 years on 425 beds. A code frame was provided. Errors were classified according to type, causes and time of detection. The most often drug implicated and the error correction rate were studied. Deep causes were determined and contributing factors were listed.

RESULTS

Among 99,536 prescriptions analyzed, 2636 errors were detected (2.65 errors per 100 orders analyzed). The most common error was omission (31.49%). The most represented cause was redundancy requirement (11.34%). Antibacterials were most commonly involved (224 errors). Exactly 65.9% of the prescriptions were modified by physicians. Three root causes were identified: (1) configuration issues; (2) misuse; (3) design problem. Three types of contributing factors have also been detailed: economic, human and technical factors.

CONCLUSIONS

Identifying root causes has targeted three types of improvement actions: (1) software settings; (2) training of users; (3) requests for improvements. Contributing factors have to be identified to control the generated risk. Some errors related to CPOE may lead to serious side effects for the patient. That is why it is necessary to identify these errors and analyze them in order to implement improvement actions and prevention to secure the prescription.

Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

OBJECTIVE

To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys.

MATERIALS AND METHODS

Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35).

RESULTS

The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods.

DISCUSSION

Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR.

CONCLUSION

We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

Detection and characterization of usability problems in structured data entry interfaces in dentistry

BACKGROUND

Poor usability is one of the major barriers for optimally using electronic health records (EHRs). Dentists are increasingly adopting EHRs, and are using structured data entry interfaces to enter data such that the data can be easily retrieved and exchanged. Until recently, dentists have lacked a standardized terminology to consistently represent oral health diagnoses.

OBJECTIVES

In this study we evaluated the usability of a widely used EHR interface that allow the entry of diagnostic terms, using multi-faceted methods to identify problems and work with the vendor to correct them using an iterative design method.

METHODS

Fieldwork was undertaken at two clinical sites, and dental providers as subjects participated in user testing (n=32), interviews (n=36) and observations (n=24).

RESULTS

User testing revealed that only 22-41% of users were able to successfully complete a simple task of entering one diagnosis, while no user was able to complete a more complex task. We identified and characterized 24 high-level usability problems reducing efficiency and causing user errors. Interface-related problems included unexpected approaches for displaying diagnosis, lack of visibility, and inconsistent use of UI widgets. Terminology related issues included missing and mis-categorized concepts. Work domain issues involved both absent and superfluous functions. In collaboration with the vendor, each usability problem was prioritized and a timeline set to resolve the concerns.

DISCUSSION

Mixed methods evaluations identified a number of critical usability issues relating to the user interface, underlying terminology of the work domain. The usability challenges were found to prevent most users from successfully completing the tasks. Our further work we will determine if changes to the interface, terminology and work domain do result in improved usability.