Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

Evaluating and improving the usability of a mHealth platform to assess postoperative dental pain

Objectives

The use of interactive mobile health (mHealth) applications to monitor patient-reported postoperative pain outcomes is an emerging area in dentistry that requires further exploration. This study aimed to evaluate and improve the usability of an existing mHealth application.

Materials and methods

The usability of the application was assessed iteratively using a 3-phase approach, including a rapid cognitive walkthrough (Phase I), lab-based usability testing (Phase II), and in situ pilot testing (Phase III). The study team conducted Phase I, while providers and patients participated in Phase II and III.

Results

The rapid cognitive walkthrough identified 23 potential issues that could negatively impact user experience, with the majority classified as system issues. The lab-based usability testing yielded 141 usability issues.; 43% encountered by patients and 57% by dentists. Usability problems encountered during pilot testing included undelivered messages due to mobile phone carrier and service-related issues, errors in patients' phone number data entry, and problems in provider training.

Discussion

Through collaborative and iterative work with the vendor, usability issues were addressed before launching a trial to assess its efficacy.

Conclusion

The usability of the mHealth application for postoperative dental pain was remarkably improved by the iterative analysis and interdisciplinary collaboration.

Towards co-design of rehabilitation technologies: a collaborative approach to prioritize usability issues

Introduction

Early stakeholder engagement is critical to the successful development and translation of rehabilitation technologies, a pivotal step of which is usability testing with intended end-users. To this end, several methods employ end-user feedback to identify usability and implementation issues. However, the process of prioritizing identified issues seldom leverages the knowledge and expertise of the range of stakeholders who will ultimately affect the demand and supply of a device. This paper describes a novel method to prioritize end-user feedback using transdisciplinary stakeholder consultation and address it in subsequent product development. The proposed approach was demonstrated using a case study relating to the development of a novel technology for neural recovery after spinal cord injury.

Method

Feedback from five individuals with chronic spinal cord injury was collected during two-hour usability evaluation sessions with a fully functional high-fidelity system prototype. A think-aloud and semi-structured interview protocol was used with each participant to identify usability and acceptability issues relating to the system in a 3-phase approach. Phase 1 involved extracting usability issues from think-aloud and semi-structured interview data. Phase 2 involved rating the usability issues based on their significance, technical feasibility, and implementation priority by relevant internal and external stakeholders. Finally, Phase 3 involved aggregating the usability issues according to design and implementation elements to facilitate solution generation, and these solutions were then raised as action tasks for future design iterations.

Results

Sixty usability issues representing nine facets of usability were rated. Eighty percent of issues were rated to be of moderate to high significance, 83% were rated as being feasible to address, and 75% were rated as addressable using existing project resources. Fifty percent of the issues were rated to be a high priority for implementation. Evaluation of the grouped issues identified 21 tasks which were mapped to the product roadmap for integration into future design iterations.

Discussion

This paper presents a method for meaningful transdisciplinary stakeholder engagement in rehabilitation technology development that can extended to other projects. Alongside a worked example, we offer practical considerations for others seeking to co-develop rehabilitation technologies.

Benchmarking usability of patient portals in Estonia, Finland, Norway, and Sweden

INTRODUCTION

Poor usability is a barrier to widespread adoption of electronic health records (EHR). Providing good usability is especially challenging in the health care context, as there is a wide variety of patient users. Usability benchmarking is an approach for improving usability by evaluating and comparing the strength and weaknesses of systems. The main purpose of this study is to benchmark usability of patient portals across countries.

METHODS

A mixed-methods survey approach was applied to benchmark the national patient portals offering patient access to EHR in Estonia, Finland, Norway, and Sweden. These Nordic countries have similar public healthcare systems, and they are pioneers in offering patients access to EHR for several years. In a survey of 29,334 patients, both patients' quantitative ratings of usability and their qualitative descriptions of very positive and very negative peak experiences of portal use were collected.

RESULTS

The usability scores ranged from good to fair level of usability. The narratives of very positive and very negative experiences included the benefits of the patient portals and experienced usability issues. The regression analysis of results showed that very positive and negative experiences of patient portal use explain 19-35% of the variation of usability scores in the four countries. The percentage of patients who reported very positive or very negative experiences in each country was unrelated to the usability scores across countries.

CONCLUSIONS

The survey approach could be used to evaluate usability with a wide variety of users and it supported learning from comparison across the countries. The combination of quantitative and qualitative data provided an approximation of the level of the perceived usability, and identified usability issues to be improved and useful features that patients appreciate. Further work is needed to improve the comparability of the varied samples across countries.

An Online Psychological Program for Adolescents and Young Adults With Headaches: Iterative Design and Rapid Usability Testing

BACKGROUND

Headache disorders are common, debilitating health problems. Cognitive-behavioral therapy (CBT) is recommended but rarely easily available. With the use of the internet and communication technologies among youth and young adults, these individuals could be self-trained in CBT skills. There is an increasing number of internet-based interventions for headaches, but there has been little research into the usability of these interventions because evaluating usability across the intervention development life cycle is costly. We developed an internet-based CBT program, the Specialized Program for Headache Reduction (SPHERE). While developing it, we aimed to improve SPHERE through rapid usability testing cycles.

OBJECTIVE

This study aims to presents a rapid and affordable usability testing approach that can be performed throughout the intervention development life cycle. This paper also provides evidence of the usability of SPHERE.

METHODS

We used the "think aloud" usability testing method based on Krug's approach to test user interaction within a lab setting. This was followed by a short posttest interview. We planned to test SPHERE with 3-5 participants testing the same part of the program each cycle. Both the design and development team and the research team actively participated in the usability testing process. Observers independently identified the top 3 usability issues, rated their severity, and conducted debriefing sessions to come to consensus on major issues and generate potential solutions.

RESULTS

The testing process allowed major usability issues to be identified and rectified rapidly before piloting SPHERE in a real-world context. A total of 2 cycles of testing were conducted. Of the usability issues encountered in cycles 1 and 2, a total of 68% (17/25) and 32% (12/38), respectively, were rated as major, discussed, and fixed.

CONCLUSIONS

This study shows that rapid usability testing is an essential part of the design process that improves program functionality and can be easy and inexpensive to undertake.

Towards Translation of Novel Neurorehabilitation Systems: A Practical Approach to Usability Testing

Usability testing is important for the effective translation of neurorehabilitation technologies but is often overlooked and under-reported. The aim of this paper is to present a method of collecting and analyzing usability data, using a think-aloud and semi-structured interview protocol and qualitative analysis techniques. We present a worked case study of this method with a novel neurorehabilitation system that utilizes thought-controlled robotics to partially restore lower-limb function of people with spinal cord injury (SCI). Five male participants (mean age = 32.6 years) with SCI who identified as users of related neurorehabilitation technologies completed the usability study. Video-recorded usability sessions utilized a combination of concurrent and retrospective think-aloud methods as well as semi-structured interviews. Recordings were analyzed to identify verbal and behavioral feedback from participants regarding system performance and acceptability. In total, 538 data points were logged, which were aggregated into 60 usability issues, 44 positive evaluations, and 31 strategies for improvement. The approach undertaken was novel in that we sought to not only capture usability issues but also system elements that were positively evaluated by intended users and strategies for improvement from the perspective of intended users. These observations will be used to inform the further development of the neurorehabilitation system.

Current state of dental informatics in the field of health information systems: a scoping review

Background

Over the past 50 years, dental informatics has developed significantly in the field of health information systems. Accordingly, several studies have been conducted on standardized clinical coding systems, data capture, and clinical data reuse in dentistry.

Methods

Based on the definition of health information systems, the literature search was divided into three specific sub-searches: “standardized clinical coding systems,” “data capture,” and “reuse of routine patient care data.” PubMed and Web of Science were searched for peer-reviewed articles. The review was conducted following the PRISMA-ScR protocol.

Results

A total of 44 articles were identified for inclusion in the review. Of these, 15 were related to “standardized clinical coding systems,” 15 to “data capture,” and 14 to “reuse of routine patient care data.” Articles related to standardized clinical coding systems focused on the design and/or development of proposed systems, on their evaluation and validation, on their adoption in academic settings, and on user perception. Articles related to data capture addressed the issue of data completeness, evaluated user interfaces and workflow integration, and proposed technical solutions. Finally, articles related to reuse of routine patient care data focused on clinical decision support systems centered on patient care, institutional or population-based health monitoring support systems, and clinical research.

Conclusions

While the development of health information systems, and especially standardized clinical coding systems, has led to significant progress in research and quality measures, most reviewed articles were published in the US. Clinical decision support systems that reuse EDR data have been little studied. Likewise, few studies have examined the working environment of dental practitioners or the pedagogical value of using health information systems in dentistry.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12903-022-02163-9.

Usability Testing of Virtual Reality Applications-The Pilot Study

The need for objective data-driven usability testing of VR applications is becoming more tangible with the rapid development of numerous VR applications and their increased accessibility. Traditional methods of testing are too time and resource consuming and might provide results that are highly subjective. Thus, the aim of this article is to explore the possibility of automation of usability testing of VR applications by using objective features such as HMD built-in head and hands tracking, EEG sensor, video recording, and other measurable parameters in addition to automated analysis of subjective data provided in questionnaires. For this purpose, a simple VR application was created which comprised relatively easy tasks that did not generate stress for the users. Fourteen volunteers took part in the study and their signals were monitored to acquire objective automated data. At the same time the observer was taking notes of subjects’ behaviour, and their subjective opinions about the experience were recorded in a post-experiment questionnaire. The results acquired from signal monitoring and questionnaires were juxtaposed with observation and post-interview results to confirm the validity and efficacy of automated usability testing. The results were very promising, proving that automated usability testing of VR applications is potentially achievable.

Caries Risk Documentation And Prevention: eMeasures For Dental Electronic Health Records

BACKGROUND

Longitudinal patient level data available in the electronic health record (EHR) allows for the development, implementation, and validations of dental quality measures (eMeasures).

OBJECTIVE

We report the feasibility and validity of implementing two eMeasures. The eMeasures determined the proportion of patients receiving a caries risk assessment (eCRA) and corresponding appropriate risk-based preventative treatments for patients at elevated risk of caries (appropriateness of care [eAoC]) in two academic institutions and one accountable care organization, in the 2019 reporting year.

METHODS

Both eMeasures define the numerator and denominator beginning at the patient level, populations' specifications, and validated the automated queries. For eCRA, patients who completed a comprehensive or periodic oral evaluation formed the denominator, and patients of any age who received a CRA formed the numerator. The eAoC evaluated the proportion of patients at elevated caries risk who received the corresponding appropriate risk-based preventative treatments.

RESULTS

EHR automated queries identified in three sites 269,536 patients who met the inclusion criteria for receiving a CRA. The overall proportion of patients who received a CRA was 94.4% (eCRA). In eAoC, patients at elevated caries risk levels (moderate, high, or extreme) received fluoride preventive treatment ranging from 56 to 93.8%. For patients at high and extreme risk, antimicrobials were prescribed more frequently site 3 (80.6%) than sites 2 (16.7%) and 1 (2.9%).

CONCLUSION

Patient-level data available in the EHRs can be used to implement process-of-care dental eCRA and AoC, eAoC measures identify gaps in clinical practice. EHR-based measures can be useful in improving delivery of evidence-based preventative treatments to reduce risk, prevent tooth decay, and improve oral health.

Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis

BACKGROUND

Electronic health record (EHR) usability and physician task load both contribute to physician professional burnout. The association between perceived EHR usability and workload has not previously been studied at a national level. Better understanding these interactions could give further information as to the drivers of extraneous task load.

OBJECTIVE

This study aimed to determine the relationship between physician-perceived EHR usability and workload by specialty and evaluate for associations with professional burnout.

METHODS

A secondary analysis of a cross-sectional survey of US physicians from all specialties was conducted from October 2017 to March 2018. Among the 1250 physicians invited to respond to the subsurvey analyzed here, 848 (67.8%) completed it. EHR usability was assessed with the System Usability Scale (SUS; range: 0-100). Provider task load (PTL) was assessed using the mental demand, physical demand, temporal demand, and effort required subscales of the National Aeronautics and Space Administration Task Load Index (range: 0-400). Burnout was measured using the Maslach Burnout Inventory.

RESULTS

The mean scores were 46.1 (SD 22.1) for SUS and 262.5 (SD 71.7) for PTL. On multivariable analysis adjusting for age, gender, relationship status, medical specialty, practice setting, hours worked per week, and number of nights on call per week, physician-rated EHR usability was associated with PTL, with each 1-point increase in SUS score (indicating more favorable) associated with a 0.57-point decrease in PTL score (P<.001). On mediation analysis, higher SUS score was associated with lower PTL score, which was associated with lower odds of burnout.

CONCLUSIONS

A strong association was observed between EHR usability and workload among US physicians, with more favorable usability associated with less workload. Both outcomes were associated with the odds of burnout, with task load acting as a mediator between EHR usability and burnout. Improving EHR usability while decreasing task load has the potential to allow practicing physicians more working memory for medical decision making and patient communication.

A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality

Health-related virtual reality (VR) applications for patient treatment, rehabilitation, and medical professional training are on the rise. However, there is little guidance on how to select and perform usability evaluations for VR health interventions compared to the supports that exist for other digital health technologies. The purpose of this viewpoint paper is to present an introductory summary of various usability testing approaches or methods that can be used for VR applications. Along with an overview of each, a list of resources is provided for readers to obtain additionally relevant information. Six categories of VR usability evaluations are described using a previously developed classification taxonomy specific to VR environments: (1) cognitive or task walkthrough, (2) graphical evaluation, (3) post hoc questionnaires or interviews, (4) physical performance evaluation, (5) user interface evaluation, and (6) heuristic evaluation. Given the growth of VR in health care, rigorous evaluation and usability testing is crucial in the development and implementation of novel VR interventions. The approaches outlined in this paper provide a starting point for conducting usability assessments for health-related VR applications; however, there is a need to also move beyond these to adopt those from the gaming industry, where assessments for both usability and user experience are routinely conducted.

Bedside Usability Surveys: Adding Authentic User Detail to the Assessment

Errors in healthcare are a leading cause of death in the United States. Equipment usability and user interfaces remain an area not fully elucidated. Infusion pumps play a vital role in care delivery, often essential for critical therapies. While pump function is comparatively simple, their programming, configuration, and form provide opportunity for error. Our purpose was to assess qualities nurses identified as important to pump operation by electronic survey. A self-developed usability survey was distributed to a random sample of 500 nurses, stratified by unit type and employed at the large academic quaternary care hospital. The overall response rate was 48% (n = 240). Descriptive and inferential statistics describe the responses and represent more than 5500 weekly infusions. Nurses described confident use of the system with some differences by unit type. Ninety percent of respondents indicated they have omitted use of the dose error reduction system, which should raise safety concerns. Users reported issues with the user interface and error prevention systems. Qualitative items elicited suggestions for improving aspects of the pump. Employing a usability survey in a clinical area proved to be a simple, inexpensive way to gather more information on the use and potential improvements of infusion pumps.

A micro-analytic approach to understanding electronic health record navigation paths

Clinician task performance is significantly impacted by the navigational efficiency of the system interface. Here we propose and evaluate a navigational complexity framework useful for examining differences in electronic health record (EHR) interface systems and their impact on task performance. The methodological approach includes 1) expert-based methods-specifically, representational analysis (focused on interface elements), keystroke level modeling (KLM), and cognitive walkthrough; and 2) quantitative analysis of interactive behaviors based on video-captured observations. Medication administration record (MAR) tasks completed by nurses during preoperative (PreOp) patient assessment were studied across three Mayo Clinic regional campuses and three different EHR systems. By analyzing the steps executed within the interfaces involved to complete the MAR tasks, we characterized complexities in EHR navigation. These complexities were reflected in time spent on task, click counts, and screen transitions, and were found to potentially influence nurses' performance. Two of the EHR systems, employing a single screen format, required less time to complete (mean 101.5, range 106-97 s), respectively, compared to one system employing multiple screens (176 s, 73% increase). These complexities surfaced through trade-offs in cognitive processes that could potentially influence nurses' performance. Factors such as perceptual-motor activity, visual search, and memory load impacted navigational complexity. An implication of this work is that small tractable changes in interface design can substantially improve EHR navigation, overall usability, and workflow.

Speech recognition for clinical documentation from 1990 to 2018: a systematic review

OBJECTIVE

The study sought to review recent literature regarding use of speech recognition (SR) technology for clinical documentation and to understand the impact of SR on document accuracy, provider efficiency, institutional cost, and more.

MATERIALS AND METHODS

We searched 10 scientific and medical literature databases to find articles about clinician use of SR for documentation published between January 1, 1990, and October 15, 2018. We annotated included articles with their research topic(s), medical domain(s), and SR system(s) evaluated and analyzed the results.

RESULTS

One hundred twenty-two articles were included. Forty-eight (39.3%) involved the radiology department exclusively and 10 (8.2%) involved emergency medicine; 10 (8.2%) mentioned multiple departments. Forty-eight (39.3%) articles studied productivity; 20 (16.4%) studied the effect of SR on documentation time, with mixed findings. Decreased turnaround time was reported in all 19 (15.6%) studies in which it was evaluated. Twenty-nine (23.8%) studies conducted error analyses, though various evaluation metrics were used. Reported percentage of documents with errors ranged from 4.8% to 71%; reported word error rates ranged from 7.4% to 38.7%. Seven (5.7%) studies assessed documentation-associated costs; 5 reported decreases and 2 reported increases. Many studies (44.3%) used products by Nuance Communications. Other vendors included IBM (9.0%) and Philips (6.6%); 7 (5.7%) used self-developed systems.

CONCLUSION

Despite widespread use of SR for clinical documentation, research on this topic remains largely heterogeneous, often using different evaluation metrics with mixed findings. Further, that SR-assisted documentation has become increasingly common in clinical settings beyond radiology warrants further investigation of its use and effectiveness in these settings.

Higher Education Students Perspective on Education Management Information Systems

As higher education evolves into a multifaceted and complex activity, the incorporation of education management information systems (EMIS) that allows for the production of relevant, organized and structured information, becomes a necessity for both institutions and students. Despite the recognition of this requirement, existing literature does not focus on how EMIS might trigger students' success. With this in mind, an initial proposal of a multi-perspective EMIS success model is presented and a validation on the possible existence of linear correlations between the model contexts is described. Moderate correlations have been detected between the majority of the model contexts and a very strong correlation has been detected between students' satisfaction and the arise of net benefits associated with the use of EMIS.

Too much or too little? Investigating the usability of high and low data displays of the same electronic medical record

The high data density on electronic medical record screens is touted as a major usability issue. However, it may not be a problem if the data is relevant and well-organized. Our objective was to test this assumption using a comprehensive set of measures that assess the three pillars of usability: efficiency (both physical and cognitive), effectiveness, and satisfaction. Physicians were asked to go through a series of tasks using two versions of the same electronic medical record: one where all the display items were separated into tabs (the original display), and one where important display items were grouped logically in one tab (the redesigned display). Results supported the hypothesis that combining relevant data in organized fashion into a smaller location would improve usability. The findings highlight the role of good display organization to mitigate the effects of high data density, as well as the importance of assessing cognitive load as part of usability studies.

Human-centered design of a personal health record system for metabolic syndrome management based on the ISO 9241-210:2010 standard

Background

Previous studies have demonstrated the effectiveness of information and communication technologies to support healthy lifestyle interventions. In particular, personal health record systems (PHR-Ss) empower self-care, essential to support lifestyle changes. Approaches such as the user-centered design (UCD), which is already a standard within the software industry (ISO 9241-210:2010), provide specifications and guidelines to guarantee user acceptance and quality of eHealth systems. However, no single PHR-S for metabolic syndrome (MS) developed following the recommendations of the ISO 9241-210:2010 specification has been found in the literature.

Objective

The aim of this study was to describe the development of a PHR-S for the management of MS according to the principles and recommendations of the ISO 9241-210 standard.

Methods

The proposed PHR-S was developed using a formal software development process which, in addition to the traditional activities of any software process, included the principles and recommendations of the ISO 9241-210 standard. To gather user information, a survey sample of 1,187 individuals, eight interviews, and a focus group with seven people were performed. Throughout five iterations, three prototypes were built. Potential users of each system evaluated each prototype. The quality attributes of efficiency, effectiveness, and user satisfaction were assessed using metrics defined in the ISO/IEC 25022 standard.

Results

The following results were obtained: 1) a technology profile from 1,187 individuals at risk for MS from the city of Popayan, Colombia, identifying that 75.2% of the people use the Internet and 51% had a smartphone; 2) a PHR-S to manage MS developed (the PHR-S has the following five main functionalities: record the five MS risk factors, share these measures with health care professionals, and three educational modules on nutrition, stress management, and a physical activity); and 3) usability tests on each prototype obtaining the following results: 100% effectiveness, 100% efficiency, and 84.2 points in the system usability scale.

Conclusion

The software development methodology used was based on the ISO 9241-210 standard, which allowed the development team to maintain a focus on user’s needs and requirements throughout the project, which resulted in an increased satisfaction and acceptance of the system. Additionally, the establishment of a multidisciplinary team allowed the application of considerations not only from the disciplines of software engineering and health sciences but also from other disciplines such as graphical design and media communication. Finally, usability testing allowed the observation of flaws in the designs, which helped to improve the solution.

Proposing Electronic Health Record Usability Requirements Based on Enriched ISO 9241 Metric Usability Model

Introduction

System usability assessment is among the important aspects in assessing the quality of clinical information technology, especially when the end users of the system are concerned. This study aims at providing a comprehensive list of system usability.

Methods

This research is a descriptive cross-sectional one conducted using Delphi technique in three phases in 2013. After experts' ideas were concluded, the final version of the questionnaire including 163 items in three phases was presented to 40 users of information systems in hospitals. The grading ranged from 0-4. Data analysis was conducted using SPSS software. Those requirements with a mean point of three or higher were finally confirmed.

Results

The list of system usability requirements for electronic health record was designed and confirmed in nine areas including suitability for the task (24 items), self-descriptiveness (22 items), controllability (19 questions), conformity with user expectations (25 items), error tolerance (21 items), suitability for individualization (7 items), suitability for learning (19 items), visual clarity (18 items) and auditory presentation (8 items).

Conclusion

A relatively comprehensive model including useful requirements for using EHR was presented which can increase functionality, effectiveness and users' satisfaction. Thus, it is suggested that the present model be adopted by system designers and healthcare system institutions to assess those systems.

Usability Evaluation of Electronic Health Record System around Clinical Notes Usage-An Ethnographic Study

Background A substantial gap exists between current Electronic Health Record (EHR) usability and potential optimal usability. One of the fundamental reasons for this discrepancy is poor incorporation of a User-Centered Design (UCD) approach during the Graphical User Interface (GUI) development process. Objective To evaluate usability strengths and weaknesses of two widely implemented EHR GUIs for critical clinical notes usage tasks. Methods Twelve Internal Medicine resident physicians interacting with one of the two EHR systems (System-1 at Location-A and System-2 at Location-B) were observed by two usability evaluators employing an ethnographic approach. User comments and observer findings were analyzed for two critical tasks: (1) clinical notes entry and (2) related information-seeking tasks. Data were analyzed from two standpoints: (1) usability references categorized by usability evaluators as positive, negative, or equivocal and (2) usability impact of each feature measured through a 7-point severity rating scale. Findings were also validated by user responses to a post observation questionnaire. Results For clinical notes entry, System-1 surpassed System-2 with more positive (26% vs. 12%) than negative (12% vs. 34%) usability references. Greatest impact features on EHR usability (severity score pertaining to each feature) for clinical notes entry were: autopopulation (6), screen options (5.5), communication (5), copy pasting (4.5), error prevention (4.5), edit ability (4), and dictation and transcription (3.5). Both systems performed equally well on information-seeking tasks and features with greatest impacts on EHR usability were navigation for notes (7) and others (e.g., looking for ancillary data; 5.5). Ethnographic observations were supported by follow-up questionnaire responses. Conclusion This study provides usability-specific insights to inform future, improved, EHR interface that is better aligned with UCD approach.

Human Factors and Ergonomics in the Design of Health Information Technology: Trends and Progress in 2014

OBJECTIVE

To summarize significant contributions to the research on human factors and organizational issues in medical informatics.

METHODS

An extensive search using PubMed/Medline and Web of Science® was conducted to identify the scientific contributions, published in 2014, to human factors and organizational issues in medical informatics, with a focus on health information technology (HIT) usability. The selection process comprised three steps: (i) 15 candidate best papers were selected by the two section editors, (ii) external reviewers from a pool of international experts reviewed each candidate best paper, and (iii) the final selection of three best papers was made by the editorial board of the IMIA Yearbook.

RESULTS

Noteworthy papers published in 2014 describe an efficient, easy to implement, and useful process for detecting and mitigating human factors and ergonomics (HFE) issues of HIT. They contribute to promote the HFE approach with interventions based on rigorous and well-conducted methods when designing and implementing HIT.

CONCLUSION

The application of HFE in the design and implementation of HIT remains limited, and the impact of incorporating HFE principles on patient safety is understudied. Future works should be conducted to advance this field of research, so that the safety and quality of patient care are not compromised by the increasing adoption of HIT.

Strategic Shift to a Diagnostic Model of Care in a Multi-Site Group Dental Practice

Background

Documenting standardized dental diagnostic terms represents an emerging change for how dentistry is practiced. We focused on a mid-sized dental group practice as it shifted to a policy of documenting patients’ diagnoses using standardized terms in the electronic health record.

Methods

Kotter’s change framework was translated into interview questions posed to the senior leadership in a mid-size dental group practice. In addition, quantitative content analyses were conducted on the written policies and forms before and after the implementation of standardized diagnosis documentation to assess the extent to which the forms and policies reflected the shift. Three reviewers analyzed the data individually and reached consensuses where needed.

Results

Kotter’s guiding change framework explained the steps taken to 97 percent utilization rate of the Electronic Health Record and Dental Diagnostic Code. Of the 96 documents included in the forms and policy analysis, 31 documents were officially updated but only two added a diagnostic element.

Conclusion

Change strategies established in the business literature hold utility for dental practices seeking diagnosis-centered care.

Practical Implications

A practice that shifts to a diagnosis-driven care philosophy would be best served by ensuring that the change process follows a leadership framework that is calibrated to the organization’s culture.

Preprocessing structured clinical data for predictive modeling and decision support. A roadmap to tackle the challenges

BACKGROUND

EHR systems have high potential to improve healthcare delivery and management. Although structured EHR data generates information in machine-readable formats, their use for decision support still poses technical challenges for researchers due to the need to preprocess and convert data into a matrix format. During our research, we observed that clinical informatics literature does not provide guidance for researchers on how to build this matrix while avoiding potential pitfalls.

OBJECTIVES

This article aims to provide researchers a roadmap of the main technical challenges of preprocessing structured EHR data and possible strategies to overcome them.

METHODS

Along standard data processing stages - extracting database entries, defining features, processing data, assessing feature values and integrating data elements, within an EDPAI framework -, we identified the main challenges faced by researchers and reflect on how to address those challenges based on lessons learned from our research experience and on best practices from related literature. We highlight the main potential sources of error, present strategies to approach those challenges and discuss implications of these strategies.

RESULTS

Following the EDPAI framework, researchers face five key challenges: (1) gathering and integrating data, (2) identifying and handling different feature types, (3) combining features to handle redundancy and granularity, (4) addressing data missingness, and (5) handling multiple feature values. Strategies to address these challenges include: cross-checking identifiers for robust data retrieval and integration; applying clinical knowledge in identifying feature types, in addressing redundancy and granularity, and in accommodating multiple feature values; and investigating missing patterns adequately.

CONCLUSIONS

This article contributes to literature by providing a roadmap to inform structured EHR data preprocessing. It may advise researchers on potential pitfalls and implications of methodological decisions in handling structured data, so as to avoid biases and help realize the benefits of the secondary use of EHR data.

User-centered design of multi-gene sequencing panel reports for clinicians

The objective of this study was to develop a high-fidelity prototype for delivering multi-gene sequencing panel (GS) reports to clinicians that simulates the user experience of a final application. The delivery and use of GS reports can occur within complex and high-paced healthcare environments. We employ a user-centered software design approach in a focus group setting in order to facilitate gathering rich contextual information from a diverse group of stakeholders potentially impacted by the delivery of GS reports relevant to two precision medicine programs at the University of Maryland Medical Center. Responses from focus group sessions were transcribed, coded and analyzed by two team members. Notification mechanisms and information resources preferred by participants from our first phase of focus groups were incorporated into scenarios and the design of a software prototype for delivering GS reports. The goal of our second phase of focus group, to gain input on the prototype software design, was accomplished through conducting task walkthroughs with GS reporting scenarios. Preferences for notification, content and consultation from genetics specialists appeared to depend upon familiarity with scenarios for ordering and delivering GS reports. Despite familiarity with some aspects of the scenarios we proposed, many of our participants agreed that they would likely seek consultation from a genetics specialist after viewing the test reports. In addition, participants offered design and content recommendations. Findings illustrated a need to support customized notification approaches, user-specific information, and access to genetics specialists with GS reports. These design principles can be incorporated into software applications that deliver GS reports. Our user-centered approach to conduct this assessment and the specific input we received from clinicians may also be relevant to others working on similar projects.

Integrating data from an online diabetes prevention program into an electronic health record and clinical workflow, a design phase usability study

Background

Health information is increasingly being digitally stored and exchanged. The public is regularly collecting and storing health-related data on their own electronic devices and in the cloud. Diabetes prevention is an increasingly important preventive health measure, and diet and exercise are key components of this. Patients are turning to online programs to help them lose weight. Despite primary care physicians being important in patients’ weight loss success, there is no exchange of information between the primary care provider (PCP) and these online weight loss programs. There is an emerging opportunity to integrate this data directly into the electronic health record (EHR), but little is known about what information to share or how to share it most effectively. This study aims to characterize the preferences of providers concerning the integration of externally generated lifestyle modification data into a primary care EHR workflow.

Methods

We performed a qualitative study using two rounds of semi-structured interviews with primary care providers. We used an iterative design process involving primary care providers, health information technology software developers and health services researchers to develop the interface.

Results

Using grounded-theory thematic analysis 4 themes emerged from the interviews: 1) barriers to establishing healthy lifestyles, 2) features of a lifestyle modification program, 3) reporting of outcomes to the primary care provider, and 4) integration with primary care. These themes guided the rapid-cycle agile design process of an interface of data from an online diabetes prevention program into the primary care EHR workflow.

Conclusions

The integration of external health-related data into the EHR must be embedded into the provider workflow in order to be useful to the provider and beneficial for the patient. Accomplishing this requires evaluation of that clinical workflow during software design. The development of this novel interface used rapid cycle iterative design, early involvement by providers, and usability testing methodology. This provides a framework for how to integrate external data into provider workflow in efficient and effective ways. There is now the potential to realize the importance of having this data available in the clinical setting for patient engagement and health outcomes.

Effectiveness of Implementation of Electronic Malaria Information System as the National Malaria Surveillance System in Thailand

Background

In moving toward malaria elimination, one strategy is to implement an active surveillance system for effective case management. Thailand has developed and implemented the electronic Malaria Information System (eMIS) capturing individualized electronic records of suspected or confirmed malaria cases.

Objective

The main purpose of this study was to determine how well the eMIS improves the quality of Thailand’s malaria surveillance system. In particular, the focus of the study was to evaluate the effectiveness of the eMIS in terms of the system users’ perception and the system outcomes (ie, quality of data) regarding the management of malaria patients.

Methods

A mixed-methods technique was used with the framework based on system effectiveness attributes: data quality, timeliness, simplicity, acceptability, flexibility, stability, and usefulness. Three methods were utilized: data records review, survey of system users, and in-depth interviews with key stakeholders. From the two highest endemic provinces, paper forms matching electronic records of 4455 noninfected and 784 malaria-infected cases were reviewed. Web-based anonymous questionnaires were distributed to all 129 eMIS data entry staff throughout Thailand, and semistructured interviews were conducted with 12 management-level officers.

Results

The eMIS is well accepted by system users at both management and operational levels. The data quality has enabled malaria personnel to perform more effective prevention and control activities. There is evidence of practices resulting in inconsistencies and logical errors in data reporting. Critical data elements were mostly completed, except for a few related to certain dates and area classifications. Timeliness in reporting a case to the system was acceptable with a delay of 3-4 days. The evaluation of quantitative and qualitative data confirmed that the eMIS has high levels of simplicity, acceptability, stability, and flexibility.

Conclusions

Overall, the system implemented has achieved its objective. The results of the study suggested that the eMIS helps improve the quality of Thailand’s malaria surveillance system. As the national malaria surveillance system, the eMIS’s functionalities have provided the malaria staff working at the point of care with close-to-real-time case management data quality, covering case detection, case investigation, drug compliance, and follow-up visits. Such features has led to an improvement in the quality of the malaria control program; the government officials now have quicker access to both individual and aggregated data to promptly react to possible outbreak. The eMIS thus plays one of the key roles in moving toward the national goal of malaria elimination by the next decade.

Assessing and comparing the usability of Chinese EHRs used in two Peking University hospitals to EHRs used in the US: A method of RUA

OBJECTIVES

The objective of this study was to investigate the usability level of Chinese hospital Electronic Health Records (EHRs) by assessing the completion times of EHRs for seven "meaningful use (MU)" relevant tasks conducted at two Chinese tertiary hospitals and comparing the results to those of relevant research conducted in US EHRs.

METHODS

Using Rapid Usability Assessment (RUA) developed by the National Center for Cognitive Informatics and Decision Making (NCCD), the usability of EHRs from two Peking University hospitals was assessed using a three-step Keystroke Level Model (KLM) in a laboratory environment.

RESULTS

(1) The total EHR task completion time for 7 MU relevant test tasks showed no significant differences between the two Chinese EHRs and their US counterparts, in which the time for thinking was relatively large and comprised 35.6% of the total time. The time for the electronic physician order was the largest. (2) For specific tasks, the mean completion times of the 2 hospital EHR systems spent on recording, modifying and searching (RMS) the medication orders were similar to those for the RMS radioactive tests; the mean time spent on the RMS laboratory test orders were much less. (3) There were 85 usability problems identified in the 2 hospital EHR systems.

DISCUSSION

In Chinese EHRs, a substantial amount of time is required to complete tasks relevant to MU targets and many preventable usability problems can be discovered. The task completion time of the 2 Chinese EHR systems was a little shorter than in the 5 reported US EHR systems, while the differences in smoking status and CPOE tasks were obvious; one main reason for these differences was the use of structured data entry.

CONCLUSIONS

The efficiency of Chinese and US EHRs was not significantly different. The key to improving the efficiency of both systems lies in expediting the Computerized physician order entry (CPOE) task. Many usability problems can be identified using heuristic assessments and improved by corresponding actions.

Periodontal Management by Risk Assessment: A Pragmatic Approach

An evidence-based periodontal disease risk assessment and diagnosis system has been developed and combined with a clinical decision support and management program to improve treatment and measure patient outcomes.

BACKGROUND

There is little agreement on a universally accepted periodontal risk assessment, periodontal diagnosis, and treatment management tool and their incorporation into dental practice to improve patient care. This article highlights the development and use of a practical periodontal management and risk assessment program that can be implemented in dental settings.

METHODS

The approach taken by Willamette Dental Group to develop a periodontal disease risk assessment, periodontal diagnosis, and treatment management tool is described using evidence-based best practices. With goals of standardized treatment interventions while maintaining personalized care and improved communication, this process is described to facilitate its incorporation into other dental settings.

CONCLUSIONS

Current electronic health records can be leveraged to enhance patient-centered care through the use of risk assessments and standardized guidelines to more effectively assess, diagnose, and treat patients to improve outcomes. Dental hygienists, and other committed providers, with their emphasis on prevention of periodontal disease can be principal drivers in creation and implementation of periodontal risk assessments and personalized treatment planning. Willamette Dental Group believes that such evidence-based tools can advance dentistry to new diagnostic and treatment standards.

Examining perceptions of the usefulness and usability of a mobile-based system for pharmacogenomics clinical decision support: a mixed methods study

Background. Pharmacogenomic testing has the potential to improve the safety and efficacy of pharmacotherapy, but clinical application of pharmacogenetic knowledge has remained uncommon. Clinical Decision Support (CDS) systems could help overcome some of the barriers to clinical implementation. The aim of this study was to evaluate the perception and usability of a web- and mobile-enabled CDS system for pharmacogenetics-guided drug therapy–the Medication Safety Code (MSC) system–among potential users (i.e., physicians and pharmacists). Furthermore, this study sought to collect data on the practicability and comprehensibility of potential layouts of a proposed personalized pocket card that is intended to not only contain the machine-readable data for use with the MSC system but also human-readable data on the patient’s pharmacogenomic profile. Methods. We deployed an emergent mixed methods design encompassing (1) qualitative interviews with pharmacists and pharmacy students, (2) a survey among pharmacogenomics experts that included both qualitative and quantitative elements and (3) a quantitative survey among physicians and pharmacists. The interviews followed a semi-structured guide including a hypothetical patient scenario that had to be solved by using the MSC system. The survey among pharmacogenomics experts focused on what information should be printed on the card and how this information should be arranged. Furthermore, the MSC system was evaluated based on two hypothetical patient scenarios and four follow-up questions on the perceived usability. The second survey assessed physicians’ and pharmacists’ attitude towards the MSC system. Results. In total, 101 physicians, pharmacists and PGx experts coming from various relevant fields evaluated the MSC system. Overall, the reaction to the MSC system was positive across all investigated parameters and among all user groups. The majority of participants were able to solve the patient scenarios based on the recommendations displayed on the MSC interface. A frequent request among participants was to provide specific listings of alternative drugs and concrete dosage instructions. Negligence of other patient-specific factors for choosing the right treatment such as renal function and co-medication was a common concern related to the MSC system, while data privacy and cost-benefit considerations emerged as the participants’ major concerns regarding pharmacogenetic testing in general. The results of the card layout evaluation indicate that a gene-centered and tabulated presentation of the patient’s pharmacogenomic profile is helpful and well-accepted. Conclusions. We found that the MSC system was well-received among the physicians and pharmacists included in this study. A personalized pocket card that lists a patient’s metabolizer status along with critically affected drugs can alert physicians and pharmacists to the availability of essential therapy modifications.

Drivers Advancing Oral Health in a Large Group Dental Practice Organization

Three change drivers are being implemented to high standards of patient centric and evidence-based oral health care within the context of a large multispecialty dental group practice organization based on the commitment of the dental hygienist chief operating officer and her team.

BACKGROUND AND PURPOSE

A recent environmental scan elucidated 6 change drivers that can impact the provision of oral health care. Practitioners who can embrace and maximize aspects of these change drivers will move dentistry forward and create future opportunities. This article explains how 3 of these change drivers are being applied in a privately held, accountable risk-bearing entity that provides individualized treatment programs for more than 417,000 members. To facilitate integration of the conceptual changes related to the drivers, a multi-institutional, multidisciplinary, highly functioning collaborative work group was formed.

METHODS AND APPROACH

The document Dental Hygiene at a Crossroads for Change(1) inspired the first author, a dental hygienist in a unique position as chief operating officer of a large group practice, to pursue evidence-based organizational change and to impact the quality of patient care. This was accomplished by implementing technological advances including dental diagnosis terminology in the electronic health record, clinical decision support, standardized treatment guidelines, quality metrics, and patient engagement to improve oral health outcomes at the patient and population levels. The systems and processes used to implement 3 change drivers into a large multi-practice dental setting is presented to inform and inspire others to implement change drivers with the potential for advancing oral health.

CONCLUSIONS

Technology implementing best practices and improving patient engagement are excellent drivers to advance oral health and are an effective use of oral health care dollars. Improved oral health can be leveraged through technological advances to improve clinical practice.

Measuring up: Implementing a dental quality measure in the electronic health record context

BACKGROUND

Quality improvement requires using quality measures that can be implemented in a valid manner. Using guidelines set forth by the Meaningful Use portion of the Health Information Technology for Economic and Clinical Health Act, the authors assessed the feasibility and performance of an automated electronic Meaningful Use dental clinical quality measure to determine the percentage of children who received fluoride varnish.

METHODS

The authors defined how to implement the automated measure queries in a dental electronic health record. Within records identified through automated query, the authors manually reviewed a subsample to assess the performance of the query.

RESULTS

The automated query results revealed that 71.0% of patients had fluoride varnish compared with the manual chart review results that indicated 77.6% of patients had fluoride varnish. The automated quality measure performance results indicated 90.5% sensitivity, 90.8% specificity, 96.9% positive predictive value, and 75.2% negative predictive value.

CONCLUSIONS

The authors' findings support the feasibility of using automated dental quality measure queries in the context of sufficient structured data. Information noted only in free text rather than in structured data would require using natural language processing approaches to effectively query electronic health records.

PRACTICAL IMPLICATIONS

To participate in self-directed quality improvement, dental clinicians must embrace the accountability era. Commitment to quality will require enhanced documentation to support near-term automated calculation of quality measures.

Usability, learnability and performance evaluation of Intelligent Research and Intervention Software: A delivery platform for eHealth interventions

Evaluation of an eHealth platform, Intelligent Research and Intervention Software was undertaken via cross-sectional survey of staff users and application performance monitoring. The platform is used to deliver psychosocial interventions across a range of clinical contexts, project scopes, and delivery modalities (e.g. hybrid telehealth, fully online self-managed, randomized control trials, and clinical service delivery). Intelligent Research and Intervention Software supports persuasive technology elements (e.g. tailoring, reminders, and personalization) as well as staff management tools. Results from the System Usability Scale involving 30 Staff and Administrative users across multiple projects were positive with overall mean score of 70 ("Acceptable"). The mean score for "Usability" sub-scale was 82 and for "Learnability" sub-scale 61. There were no significant differences in perceptions of usability across user groups or levels of experience. Application performance management analytics (e.g. Application Performance Index scores) across two test sites indicate the software platform is robust and reliable when compared to industry standards. Intelligent Research and Intervention Software is successfully operating as a flexible platform for creating, delivering, and evaluating eHealth interventions.

Telerehabilitation store and forward applications: a review of applications and privacy considerations in physical and occupational therapy practice

An overview of store and forward applications commonly used in physical and occupational therapy practice is reviewed with respect to regulation, privacy, security, and clinical applications. A privacy and security checklist provides a clear reference of pertinent regulatory issues regarding these software applications. A case study format is used to highlight clinical applications of store and forward software features. Important considerations of successful implementation of store and forward applications are also identified and discussed.

Mobile PHRs compliance with Android and iOS usability guidelines

Mobile Personal Health Records (PHRs) have achieved a particularly strong market share since the appearance of more powerful mobile devices and popular worldwide mobile application markets such as Apple's App Store and Android's Google Play. However, Android and Apple have a set of recommendations on design and usability targeted towards developers who wish to publish apps in their stores: Android Design Guidelines and iOS Human Interface Guidelines. This paper aims to evaluate compliance with these guidelines by assessing the usability recommendations of a set of 24 selected mobile PHR applications. An analysis process based on a well-known Systematic Literature Review (SLR) protocol was used. The results show that the 24 mobile PHR applications studied are not suitably structured. 46 % of these applications do not use any of the recommended patterns, using instead lists or springboards, which are deprecated patterns for top-level menus. 70 % of the PHRs require a registration to be able to test the application when these interactions should be delayed. Our study will help both PHR users to select user-friendly mobile PHRs and PHR providers and developers to identify the good usability practices implemented by the applications with the highest scores.