Usability Flaws in Medication Alerting Systems: Impact on Usage and Work System

Can the integration of new rules into a clinical decision support system reduce the incidence of acute kidney injury and hyperkalemia among hospitalized older adults: a protocol for a stepped-wedge, cluster-randomized trial (DETECT-IP)

BACKGROUND

Clinical decision support systems (CDSSs) enable the automated, real-time detection of situations associated with a risk of adverse drug events (ADEs). However, the effectiveness of CDSS in reducing ADEs has yet to be demonstrated. We have chosen to focus on the detection of ADE such as hyperkalemia and/or acute kidney injury (AKI), which are common among hospitalized older adults. The present study's primary objective is to use a CDSS to reduce the number of ADEs (such as AKI and/or hyperkalemia) that occur in hospitalized older adults.

METHODS

This is a multicenter, stepped-wedge, cluster-randomized study involving five hospitals. Each hospital will start with a control period (i.e., routine care, during which each center's CDSS is deactivated) and then switch to an intervention period (during which the CDSS is activated). The intervention will be the use of a CDSS and a strategy for managing and transmitting alerts to clinical pharmacists. The rules concerning AKI and hyperkalemia have been drafted and reviewed by a multidisciplinary group. Each rule created in the CDSS is associated with a standardized procedure, based on a review of the literature. Older patients (aged 65 or over) admitted to a participating general medicine ward, a surgical ward, or obstetrics ward will be eligible for inclusion after the provision of verbal informed consent.

DISCUSSION

This study will assess the effectiveness of the CDSS in reducing the incidence of AKI and hyperkalemia. The implementation of the CDSS can assist clinical pharmacists in their daily work and is expected to prevent ADEs.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT05923983. Registered February 02, 2023.

Qualitative study of challenges with recruitment of hospitals into a cluster controlled trial of clinical decision support in Australia

OBJECTIVE

To identify barriers to hospital participation in controlled cluster trials of clinical decision support (CDS) and potential strategies for addressing barriers.

DESIGN

Qualitative descriptive design comprising semistructured interviews.

SETTING

Five hospitals in New South Wales and one hospital in Queensland, Australia.

PARTICIPANTS

Senior hospital staff, including department directors, chief information officers and those working in health informatics teams.

RESULTS

20 senior hospital staff took part. Barriers to hospital-level recruitment primarily related to perceptions of risk associated with not implementing CDS as a control site. Perceived risks included reductions in patient safety, reputational risk and increased likelihood that benefits would not be achieved following electronic medical record (EMR) implementation without CDS alerts in place. Senior staff recommended clear communication of trial information to all relevant stakeholders as a key strategy for boosting hospital-level participation in trials.

CONCLUSION

Hospital participation in controlled cluster trials of CDS is hindered by perceptions that adopting an EMR without CDS is risky for both patients and organisations. The improvements in safety expected to follow CDS implementation makes it challenging and counterintuitive for hospitals to implement EMR without incorporating CDS alerts for the purposes of a research trial. To counteract these barriers, clear communication regarding the evidence base and rationale for a controlled trial is needed.

Comparison of the validity, perceived usefulness, and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability

OBJECTIVE

Two tools are currently available in the literature to evaluate the usability of medication alert systems, the instrument for evaluating human factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aimed to compare their convergent validity, perceived usability, usefulness, strengths, and weaknesses, as well as users' preferences.

METHOD

To evaluate convergent validity, two experts mapped TEMAS' items against I-MeDeSA's items with respect to the usability dimensions they target. To assess perceived usability, usefulness, strengths, and weaknesses of both tools, staff with expertise in their medication alerting system were asked to use French versions of the TEMAS and I-MeDeSA. After the use of each tool, participants were asked to complete the System Usability Scale (SUS) and answer questions about the understandability and usefulness of each tool. Finally, participants were asked to name their preferred tool. Numeric scores were statistically compared. Free-text responses were analyzed using an inductive approach.

RESULTS

Forty-five participants from 10 hospitals took part in the study. In terms of convergent validity, I-MeDeSA focuses more on the usability of the graphical user interface while TEMAS considers a wider range of usability principles. Both tools have a fair level of perceived usability (I-MeDeSA' SUS score = 61.85 and TEMAS' SUS score = 62.87), but results highlight that revisions are necessary to both tools to improve their usability. Participants found TEMAS more useful than I-MeDeSA (t = -3.63, p =.005) and had a clear preference for TEMAS to identify problems in formative evaluation (39 of 45; 0.867, p <.001) and to compare the usability of alert systems during the procurement process (36 of 45; 0.8, p <.001).

CONCLUSIONS

The TEMAS is perceived as more useful and is preferred by participants. The I-MeDeSA seems more relevant for quick evaluations that focus on the graphical user interface. The TEMAS seems to be more suitable for in-depth usability evaluations of alert systems. Even if both tools are perceived to be equally usable, they suffer from wording, instructional, and organizational problems that hinder their use. The results of this study will be used to improve the design of I-MeDeSA and TEMAS.

Evaluating the Usability of an Emergency Department After Visit Summary: Staged Heuristic Evaluation

BACKGROUND

Heuristic evaluations, while commonly used, may inadequately capture the severity of identified usability issues. In the domain of health care, usability issues can pose different levels of risk to patients. Incorporating diverse expertise (eg, clinical and patient) in the heuristic evaluation process can help assess and address potential negative impacts on patient safety that may otherwise go unnoticed. One document that should be highly usable for patients-with the potential to prevent adverse outcomes-is the after visit summary (AVS). The AVS is the document given to a patient upon discharge from the emergency department (ED), which contains instructions on how to manage symptoms, medications, and follow-up care.

OBJECTIVE

This study aims to assess a multistage method for integrating diverse expertise (ie, clinical, an older adult care partner, and health IT) with human factors engineering (HFE) expertise in the usability evaluation of the patient-facing ED AVS.

METHODS

We conducted a three-staged heuristic evaluation of an ED AVS using heuristics developed for use in evaluating patient-facing documentation. In stage 1, HFE experts reviewed the AVS to identify usability issues. In stage 2, 6 experts of varying expertise (ie, emergency medicine physicians, ED nurses, geriatricians, transitional care nurses, and an older adult care partner) rated each previously identified usability issue on its potential impact on patient comprehension and patient safety. Finally, in stage 3, an IT expert reviewed each usability issue to identify the likelihood of successfully addressing the issue.

RESULTS

In stage 1, we identified 60 usability issues that violated a total of 108 heuristics. In stage 2, 18 additional usability issues that violated 27 heuristics were identified by the study experts. Impact ratings ranged from all experts rating the issue as "no impact" to 5 out of 6 experts rating the issue as having a "large negative impact." On average, the older adult care partner representative rated usability issues as being more significant more of the time. In stage 3, 31 usability issues were rated by an IT professional as "impossible to address," 21 as "maybe," and 24 as "can be addressed."

CONCLUSIONS

Integrating diverse expertise when evaluating usability is important when patient safety is at stake. The non-HFE experts, included in stage 2 of our evaluation, identified 23% (18/78) of all the usability issues and, depending on their expertise, rated those issues as having differing impacts on patient comprehension and safety. Our findings suggest that, to conduct a comprehensive heuristic evaluation, expertise from all the contexts in which the AVS is used must be considered. Combining those findings with ratings from an IT expert, usability issues can be strategically addressed through redesign. Thus, a 3-staged heuristic evaluation method offers a framework for integrating context-specific expertise efficiently, while providing practical insights to guide human-centered design.

Comparison of the validity, perceived usefulness and usability of I-MeDeSA and TEMAS, two tools to evaluate alert system usability: a study protocol

INTRODUCTION

Research has shown that improvements to the usability of medication alert systems are needed. For designers and decisions-makers to assess usability of their alert systems, two paper-based tools are currently available: the instrument for evaluating human-factors principles in medication-related decision support alerts (I-MeDeSA) and the tool for evaluating medication alerting systems (TEMAS). This study aims to compare the validity, usability and usefulness of both tools to identify their strengths and limitations and assist designers and decision-makers in making an informed decision about which tool is most suitable for assessing their current or prospective system.

METHODS AND ANALYSIS

First, TEMAS and I-MeDeSA will be translated into French. This translation will be validated by three experts in human factors. Then, in 12 French hospitals with a medication alert system in place, staff with expertise in the system will evaluate their alert system using the two tools successively. After the use of each tool, participants will be asked to fill in the System Usability Scale (SUS) and complete a survey on the understandability and perceived usefulness of each tool. Following the completion of both assessments, participants will be asked to nominate their preferred tool and relay their opinions on the tools. The design philosophy of TEMAS and I-MeDeSA differs on the calculation of a score, impacting the way the comparison between the tools can be performed. Convergent validity will be evaluated by matching the items of the two tools with respect to the usability dimensions they assess. SUS scores and answers to the survey will be statistically compared for I-MeDeSA and TEMAS to identify differences. Free-text responses in surveys will be analysed using an inductive approach.

ETHICS AND DISSEMINATION

Ethical approval is not required in France for a study of this nature. The results will be published in a peer-reviewed journal.

Use of Pediatric Injectable Medicines Guidelines and Associated Medication Administration Errors: A Human Reliability Analysis

Background:

In a recent human reliability analysis (HRA) of simulated pediatric resuscitations, ineffective retrieval of preparation and administration instructions from online injectable medicines guidelines was a key factor contributing to medication administration errors (MAEs).

Objective:

The aim of the present study was to use a specific HRA to understand where intravenous medicines guidelines are vulnerable to misinterpretation, focusing on deviations from expected practice (discrepancies) that contributed to large-magnitude and/or clinically significant MAEs.

Methods:

Video recordings from the original study were reanalyzed to identify discrepancies in the steps required to find and extract information from the NHS Injectable Medicines Guide (IMG) website. These data were combined with MAE data from the same original study.

Results:

In total, 44 discrepancies during use of the IMG were observed across 180 medication administrations. Of these discrepancies, 21 (48%) were associated with an MAE, 16 of which (36% of 44 discrepancies) made a major contribution to that error. There were more discrepancies (31 in total, 70%) during the steps required to access the correct drug webpage than there were in the steps required to read this information (13 in total, 30%). Discrepancies when using injectable medicines guidelines made a major contribution to 6 (27%) of 22 clinically significant and 4 (15%) of 27 large-magnitude MAEs.

Conclusion and Relevance:

Discrepancies during the use of an online injectable medicines guideline were often associated with subsequent MAEs, including those with potentially significant consequences. This highlights the need to test the usability of guidelines before clinical use.

Applying a Human-Centered Design to Develop a Patient Prioritization Tool for a Pediatric Emergency Department: Detailed Case Study of First Iterations

BACKGROUND

Overcrowding in the emergency departments has become an increasingly significant problem. Patient triage strategies are acknowledged to help clinicians manage patient flow and reduce patients' waiting time. However, electronic patient triage systems are not developed so that they comply with clinicians' workflow.

OBJECTIVE

This case study presents the development of a patient prioritization tool (PPT) and of the related patient prioritization algorithm (PPA) for a pediatric emergency department (PED), relying on a human-centered design process.

METHODS

We followed a human-centered design process, wherein we (1) performed a work system analysis through observations and interviews in an academic hospital's PED; (2) deduced design specifications; (3) designed a mock PPT and the related PPA; and (4) performed user testing to assess the intuitiveness of the icons, the effectiveness in communicating patient priority, the fit between the prioritization model implemented and the participants' prioritization rules, and the participants' satisfaction.

RESULTS

The workflow analysis identified that the PPT interface should meet the needs of physicians and nurses, represent the stages of patient care, and contain patient information such as waiting time, test status (eg, prescribed, in progress), age, and a suggestion for prioritization. The mock-up developed gives the status of patients progressing through the PED; a strip represents the patient and the patient's characteristics, including a delay indicator that compares the patient's waiting time to the average waiting time of patients with a comparable reason for emergency. User tests revealed issues with icon intuitiveness, information gaps, and possible refinements in the prioritization algorithm.

CONCLUSIONS

The results of the user tests have led to modifications to improve the usability and usefulness of the PPT and its PPA. We discuss the value of integrating human factors into the design process for a PPT for PED. The PPT/PPA has been developed and installed in Lille University Hospital's PED. Studies are carried out to evaluate the use and impact of this tool on clinicians' situation awareness and prioritization-related cognitive load, prioritization of patients, waiting time, and patients' experience.

Human Factors and Usability for Health Information Technology: Old and New Challenges

OBJECTIVES

Despite national mandates, incentives, and other programs, the design of health information technology (IT) remains problematic and usability problems continue to be reported. This paper reviews recent literature on human factors and usability of health IT, with a specific focus on research aimed at applying human factors methods and principles to improve the actual design of health IT, its use, and associated patient and clinician outcomes.

METHODS

We reviewed recent literature on human factors and usability problems of health IT and research on human-centered design of health IT for clinicians and patients.

RESULTS

Studies continue to show usability problems of health IT experienced by multiple groups of health care professionals (e.g., physicians and nurses) as well as patients. Recent research shows that usability is influenced by both designers (e.g., IT vendors) and implementers in health care organizations, and that the application of human-centered design practices needs to be further improved and extended. We welcome emerging research on the design of health IT for teams as team-based care is increasingly implemented throughout health care.

CONCLUSIONS

Progress in the application of human factors methods and principles to the design of health IT is occurring, with important information provided on their actual impact on care processes and patient outcomes. Future research should examine the work of health IT designers and implementers, which would help to develop strategies for further embedding human factors engineering in IT design processes.

Building Usability Knowledge for Health Information Technology: A Usability-Oriented Analysis of Incident Reports

BACKGROUND

The contribution of usability flaws to patient safety issues is acknowledged but not well-investigated. Free-text descriptions of incident reports may provide useful data to identify the connection between health information technology (HIT) usability flaws and patient safety.

OBJECTIVES

This article examines the feasibility of using incident reports about HIT to learn about the usability flaws that affect patient safety. We posed three questions: (1) To what extent can we gain knowledge about usability issues from incident reports? (2) What types of usability flaws, related usage problems, and negative outcomes are reported in incidents reports? (3) What are the reported usability issues that give rise to patient safety issues?

METHODS

A sample of 359 reports from the U.S. Food and Drug Administration Manufacturer and User Facility Device Experience database was examined. Descriptions of usability flaws, usage problems, and negative outcomes were extracted and categorized. A supplementary analysis was performed on the incidents which contained the full chain going from a usability flaw up to a patient safety issue to identify the usability issues that gave rise to patient safety incidents.

RESULTS

A total of 249 reports were included. We found that incident reports can provide knowledge about usability flaws, usage problems, and negative outcomes. Thirty-six incidents report how usability flaws affected patient safety (ranging from incidents without consequence, to death) involving electronic patient scales, imaging systems, and HIT for medication management. The most significant class of involved usability flaws is related to the reliability, the understandability, and the availability of the clinical information.

CONCLUSION

Incidents reports involving HIT are an exploitable source of information to learn about usability flaws and their effects on patient safety. Results can be used to convince all stakeholders involved in the HIT system lifecycle that usability should be considered seriously to prevent patient safety incidents.

Evidence-based usability design principles for medication alerting systems

Background

Usability flaws in medication alerting systems may have a negative impact on clinical use and patient safety. In order to prevent the release of alerting systems that contain such flaws, it is necessary to provide designers and evaluators with evidence-based usability design principles. The objective of the present study was to develop a comprehensive, structured list of evidence-based usability design principles for medication alerting systems.

Methods

Nine sets of design principles for medication alerting systems were analyzed, summarized, and structured. We then matched the summarized principles with a list of usability flaws in order to determine the level of underlying evidence.

Results

Fifty-eight principles were summarized from the literature and two additional principles were defined, so that each flaw was matched with a principle. We organized the 60 summarized usability design principles into 6 meta-principles, 38 principles, and 16 sub-principles. Only 15 principles were not matched with a usability flaw. The 6 meta-principles respectively covered the improvement of the signal-to-noise ratio, the support for collaborative working, the fit with a clinician’s workflow, the data display, the transparency of the alerting system, and the actionable tools to be provided within an alert.

Conclusions

It is possible to develop an evidence-based, structured, comprehensive list of usability design principles that are specific to medication alerting systems and are illustrated by the corresponding usability flaws. This list represents an improvement over the current literature. Each principle is now associated with the best available evidence of its violation. This knowledge may help to improve the usability of medication alerting systems and, ultimately, decrease the harmful consequences of the systems’ usability flaws.

Electronic supplementary material

The online version of this article (10.1186/s12911-018-0615-9) contains supplementary material, which is available to authorized users.

Are We There Yet? Human Factors Knowledge and Health Information Technology - the Challenges of Implementation and Impact

Objective: To review the developments in human factors (HF) research on the challenges of health information technology (HIT) implementation and impact given the continuing incidence of usability problems and unintended consequences from HIT development and use. Methods: A search of PubMed/Medline and Web of Science® identified HF research published in 2015 and 2016. Electronic health records (EHRs) and patient-centred HIT emerged as significant foci of recent HF research. The authors selected prominent papers highlighting ongoing HF and usability challenges in these areas. This selective rather than systematic review of recent HF research highlights these key challenges and reflects on their implications on the future impact of HF research on HIT. Results: Research provides evidence of continued poor design, implementation, and usability of HIT, as well as technology-induced errors and unintended consequences. The paper highlights support for: (i) strengthening the evidence base on the benefits of HF approaches; (ii) improving knowledge translation in the implementation of HF approaches during HIT design, implementation, and evaluation; (iii) increasing transparency, governance, and enforcement of HF best practices at all stages of the HIT system development life cycle. Discussion and Conclusion: HF and usability approaches are yet to become embedded as integral components of HIT development, implementation, and impact assessment. As HIT becomes ever-more pervasive including with patients as end-users, there is a need to expand our conceptualisation of the problems to be addressed and the suite of tactics and strategies to be used to calibrate our pro-active involvement in its improvement.

A Survey of the Literature on Unintended Consequences Associated with Health Information Technology: 2014-2015

OBJECTIVE

To summarize recent research on unintended consequences associated with implementation and use of health information technology (health IT). Included in the review are original empirical investigations published in English between 2014 and 2015 that reported unintended effects introduced by adoption of digital interventions. Our analysis focuses on the trends of this steam of research, areas in which unintended consequences have continued to be reported, and common themes that emerge from the findings of these studies.

METHOD

Most of the papers reviewed were retrieved by searching three literature databases: MEDLINE, Embase, and CINAHL. Two rounds of searches were performed: the first round used more restrictive search terms specific to unintended consequences; the second round lifted the restrictions to include more generic health IT evaluation studies. Each paper was independently screened by at least two authors; differences were resolved through consensus development.

RESULTS

The literature search identified 1,538 papers that were potentially relevant; 34 were deemed meeting our inclusion criteria after screening. Studies described in these 34 papers took place in a wide variety of care areas from emergency departments to ophthalmology clinics. Some papers reflected several previously unreported unintended consequences, such as staff attrition and patients' withholding of information due to privacy and security concerns. A majority of these studies (71%) were quantitative investigations based on analysis of objectively recorded data. Several of them employed longitudinal or time series designs to distinguish between unintended consequences that had only transient impact, versus those that had persisting impact. Most of these unintended consequences resulted in adverse outcomes, even though instances of beneficial impact were also noted. While care areas covered were heterogeneous, over half of the studies were conducted at academic medical centers or teaching hospitals.

CONCLUSION

Recent studies published in the past two years represent significant advancement of unintended consequences research by seeking to include more types of health IT applications and to quantify the impact using objectively recorded data and longitudinal or time series designs. However, more mixed-methods studies are needed to develop deeper insights into the observed unintended adverse outcomes, including their root causes and remedies. We also encourage future research to go beyond the paradigm of simply describing unintended consequences, and to develop and test solutions that can prevent or minimize their impact.

Unintended Consequences: New Problems, New Solutions. Contributions From 2015

OBJECTIVE

To select the best of the 2015 published papers on unintended consequences of healthcare information technology (HIT).

METHOD

Literature searches in several areas of scholarship, including IT, human factors, evaluation studies, medical errors, medical informatics, and implementation science. Also, because the specific terms "unintended consequences" were not often included in abstracts and titles, a more nuanced search algorithm was developed.

RESULTS

We identified 754 papers that had some empirical research on unintended consequences of HIT. An initial screen of titles and abstracts reduced this to 171 papers of potential interest. We then further filtered out papers that did not meet the following criteria: 1) the paper had to report an original empirical investigation, and 2) the impact reported had to be not negligible, i.e., in quantitative studies, the results related to unintended consequences were statistically significant; and in qualitative studies the relevant themes emerged were prominent. This resulted in 33 papers of which 15 were selected as best paper candidates. Each of these 15 papers was then separately evaluated by four reviewers. The final selection of four papers was made jointly by the external reviewers and the two section editors.

CONCLUSIONS

There is a growing awareness of the importance of HIT's unintended consequences-be they generated by the HIT vendors, the implementation process, the consultants, the users, or most probably, some combination of the above. There has also been greater creativity in use of data sources, including secondary data (e.g., medical malpractice cases and surveys) and a wider acceptance of mixed methods to identify unintended consequences. Unfortunately, the complexity of causes mitigates the value of recommendations to avoid unwanted outcomes. Suggestions are often contentious rather than obvious, setting-specific, and not universally applicable. "Lessons learned" often take on generalized-and perhaps platitudinous-forms, such as: "plan extra time," "involve all of the stakeholders," "recognize the different needs of different units or disciplines." The greater awareness of these problems, and the increased desire to identify and eliminate them is clearly reflected in the area's growing literature. We are hopeful the topic will receive additional attention and the discipline will improve its ability to identify and address these unexpected and usually adverse outcomes.

Methods for Addressing Technology-induced Errors: The Current State

OBJECTIVES

The objectives of this paper are to review and discuss the methods that are being used internationally to report on, mitigate, and eliminate technology-induced errors.

METHODS

The IMIA Working Group for Health Informatics for Patient Safety worked together to review and synthesize some of the main methods and approaches associated with technology- induced error reporting, reduction, and mitigation. The work involved a review of the evidence-based literature as well as guideline publications specific to health informatics.

RESULTS

The paper presents a rich overview of current approaches, issues, and methods associated with: (1) safe HIT design, (2) safe HIT implementation, (3) reporting on technology-induced errors, (4) technology-induced error analysis, and (5) health information technology (HIT) risk management. The work is based on research from around the world.

CONCLUSIONS

Internationally, researchers have been developing methods that can be used to identify, report on, mitigate, and eliminate technology-induced errors. Although there remain issues and challenges associated with the methodologies, they have been shown to improve the quality and safety of HIT. Since the first publications documenting technology-induced errors in healthcare in 2005, we have seen in a short 10 years researchers develop ways of identifying and addressing these types of errors. We have also seen organizations begin to use these approaches. Knowledge has been translated into practice in a short ten years whereas the norm for other research areas is of 20 years.

Development of a Standardized Rating Tool for Drug Alerts to Reduce Information Overload

BACKGROUND

A well-known problem in current clinical decision support systems (CDSS) is the high number of alerts, which are often medically incorrect or irrelevant. This may lead to the so-called alert fatigue, an overriding of alerts, including those that are clinically relevant, and underuse of CDSS in general.

OBJECTIVES

The aim of our study was to develop and to apply a standardized tool that allows its users to evaluate the quality of system-generated drug alerts. The users' ratings can subsequently be used to derive recommendations for developing a filter function to reduce irrelevant alerts.

METHODS

We developed a rating tool for drug alerts and performed a web-based evaluation study that also included a user review of alerts. In this study the following categories were evaluated: "data linked correctly", "medically correct", "action required", "medication change", "critical alert", "information gained" and "show again". For this purpose, 20 anonymized clinical cases were randomly selected and displayed in our customized CDSS research prototype, which used the summary of product characteristics (SPC) for alert generation. All the alerts that were provided were evaluated by 13 physicians. The users' ratings were used to derive a filtering algorithm to reduce overalerting.

RESULTS

In total, our CDSS research prototype generated 399 alerts. In 98 % of all alerts, medication data were rated as linked correctly to drug information; in 93 %, the alerts were assessed as "medically correct"; 19.5 % of all alerts were rated as "show again". The interrater-agreement was, on average, 68.4 %. After the application of our filtering algorithm, the rate of alerts that should be shown again decreased to 14.8 %.

CONCLUSIONS

The new standardized rating tool supports a standardized feedback of user-perceived clinical relevance of CDSS alerts. Overall, the results indicated that physicians may consider the majority of alerts formally correct but clinically irrelevant and override them. Filtering may help to reduce overalerting and increase the specificity of a CDSS.

Interface design recommendations for computerised clinical audit and feedback: Hybrid usability evidence from a research-led system

BACKGROUND

Audit and Feedback (A&F) is a widely used quality improvement technique that measures clinicians' clinical performance and reports it back to them. Computerised A&F (e-A&F) system interfaces may consist of four key components: (1) Summaries of clinical performance; (2) Patient lists; (3) Patient-level data; (4) Recommended actions. There is a lack of evidence regarding how to best design e-A&F interfaces; establishing such evidence is key to maximising usability, and in turn improving patient safety.

AIM

To evaluate the usability of a novel theoretically-informed and research-led e-A&F system for primary care (the Performance Improvement plaN GeneratoR: PINGR).

OBJECTIVES

(1) Describe PINGR's design, rationale and theoretical basis; (2) Identify usability issues with PINGR; (3) Understand how these issues may interfere with the cognitive goals of end-users; (4) Translate the issues into recommendations for the user-centred design of e-A&F systems.

METHODS

Eight experienced health system evaluators performed a usability inspection using an innovative hybrid approach consisting of five stages: (1) Development of representative user tasks, Goals, and Actions; (2) Combining Heuristic Evaluation and Cognitive Walkthrough methods into a single protocol to identify usability issues; (3) Consolidation of issues; (4) Severity rating of consolidated issues; (5) Analysis of issues according to usability heuristics, interface components, and Goal-Action structure.

RESULTS

A final list of 47 issues were categorised into 8 heuristic themes. The most error-prone heuristics were 'Consistency and standards' (13 usability issues; 28% of the total) and 'Match between system and real world' (n=10, 21%). The recommended actions component of the PINGR interface had the most usability issues (n=21, 45%), followed by patient-level data (n=5, 11%), patient lists (n=4, 9%), and summaries of clinical performance (n=4, 9%). The most error-prone Actions across all user Goals were: (1) Patient selection from a list; (2) Data identification from a figure (both population-level and patient-level); (3) Disagreement with a system recommendation.

CONCLUSIONS

By contextualising our findings within the wider literature on health information system usability, we provide recommendations for the design of e-A&F system interfaces relating to their four key components, in addition to how they may be integrated within a system.

Clinical Information Systems - From Yesterday to Tomorrow

OBJECTIVES

To review the history of clinical information systems over the past twenty-five years and project anticipated changes to those systems over the next twenty-five years.

METHODS

Over 250 Medline references about clinical information systems, quality of patient care, and patient safety were reviewed. Books, Web resources, and the author's personal experience with developing the HELP system were also used.

RESULTS

There have been dramatic improvements in the use and acceptance of clinical computing systems and Electronic Health Records (EHRs), especially in the United States. Although there are still challenges with the implementation of such systems, the rate of progress has been remarkable. Over the next twenty-five years, there will remain many important opportunities and challenges. These opportunities include understanding complex clinical computing issues that must be studied, understood and optimized. Dramatic improvements in quality of care and patient safety must be anticipated as a result of the use of clinical information systems. These improvements will result from a closer involvement of clinical informaticians in the optimization of patient care processes.

CONCLUSIONS

Clinical information systems and computerized clinical decision support have made contributions to medicine in the past. Therefore, by using better medical knowledge, optimized clinical information systems, and computerized clinical decision, we will enable dramatic improvements in both the quality and safety of patient care in the next twenty-five years.