Vision and challenges of Evidence-Based Health Informatics: a case study of a CPOE meta-analysis

Impacts on Context Aware Systems in Evidence-Based Health Informatics: A Review

Background: The application of Context Aware Computing (CAC) can be an effective, useful, feasible, and acceptable way to advance medical research and provide health services. Methods: This review was conducted in accordance with the principles of the development of a mixed methods review and existing knowledge in the field via the Synthesis Framework for the Assessment of Health Information Technology to evaluate CAC implemented by Evidence-Based Health Informatics (EBHI). A systematic search of the literature was performed during 18 November 2021–22 January 2022 in Cochrane Library, IEEE Xplore, PUBMED, Scopus and in the clinical registry platform Clinicaltrials.gov. The author included the articles in the review if they were implemented by EBHI and concerned with CAC technologies. Results: 29 articles met the inclusion criteria and refer to 26 trials published between 2011 and 2022. The author noticed improvements in healthcare provision using EBHI in the findings of CAC application. She also confirmed that CAC systems are a valuable and reliable method in health care provision. Conclusions: The use of CAC systems in healthcare is a promising new area of research and development. The author presented that the evaluation of CAC systems in EBHI presents positive effects on the state of health and the management of long-term diseases. These implications are presented in this article in a detailed, clear, and reliable manner.

Evidence-Based Health Informatics as the Foundation for the COVID-19 Response: A Joint Call for Action

Background  As a major public health crisis, the novel coronavirus disease 2019 (COVID-19) pandemic demonstrates the urgent need for safe, effective, and evidence-based implementations of digital health. The urgency stems from the frequent tendency to focus attention on seemingly high promising digital health interventions despite being poorly validated in times of crisis.

Aim  In this paper, we describe a joint call for action to use and leverage evidence-based health informatics as the foundation for the COVID-19 response and public health interventions. Tangible examples are provided for how the working groups and special interest groups of the International Medical Informatics Association (IMIA) are helping to build an evidence-based response to this crisis.

Methods  Leaders of working and special interest groups of the IMIA, a total of 26 groups, were contacted via e-mail to provide a summary of the scientific-based efforts taken to combat COVID-19 pandemic and participate in the discussion toward the creation of this manuscript. A total of 13 groups participated in this manuscript.

Results  Various efforts were exerted by members of IMIA including (1) developing evidence-based guidelines for the design and deployment of digital health solutions during COVID-19; (2) surveying clinical informaticians internationally about key digital solutions deployed to combat COVID-19 and the challenges faced when implementing and using them; and (3) offering necessary resources for clinicians about the use of digital tools in clinical practice, education, and research during COVID-19.

Discussion  Rigor and evidence need to be taken into consideration when designing, implementing, and using digital tools to combat COVID-19 to avoid delays and unforeseen negative consequences. It is paramount to employ a multidisciplinary approach for the development and implementation of digital health tools that have been rapidly deployed in response to the pandemic bearing in mind human factors, ethics, data privacy, and the diversity of context at the local, national, and international levels. The training and capacity building of front-line workers is crucial and must be linked to a clear strategy for evaluation of ongoing experiences.

Publication and related bias in quantitative health services and delivery research: a multimethod study

Background

Bias in the publication and reporting of research findings (referred to as publication and related bias here) poses a major threat in evidence synthesis and evidence-based decision-making. Although this bias has been well documented in clinical research, little is known about its occurrence and magnitude in health services and delivery research.

Objectives

To obtain empirical evidence on publication and related bias in quantitative health services and delivery research; to examine current practice in detecting/mitigating this bias in health services and delivery research systematic reviews; and to explore stakeholders’ perception and experiences concerning such bias.

Methods

The project included five distinct but interrelated work packages. Work package 1 was a systematic review of empirical and methodological studies. Work package 2 involved a survey (meta-epidemiological study) of randomly selected systematic reviews of health services and delivery research topics (n = 200) to evaluate current practice in the assessment of publication and outcome reporting bias during evidence synthesis. Work package 3 included four case studies to explore the applicability of statistical methods for detecting such bias in health services and delivery research. In work package 4 we followed up four cohorts of health services and delivery research studies (total n = 300) to ascertain their publication status, and examined whether publication status was associated with statistical significance or perceived ‘positivity’ of study findings. Work package 5 involved key informant interviews with diverse health services and delivery research stakeholders (n = 24), and a focus group discussion with patient and service user representatives (n = 8).

Results

We identified only four studies that set out to investigate publication and related bias in health services and delivery research in work package 1. Three of these studies focused on health informatics research and one concerned health economics. All four studies reported evidence of the existence of this bias, but had methodological weaknesses. We also identified three health services and delivery research systematic reviews in which findings were compared between published and grey/unpublished literature. These reviews found that the quality and volume of evidence and effect estimates sometimes differed significantly between published and unpublished literature. Work package 2 showed low prevalence of considering/assessing publication (43%) and outcome reporting (17%) bias in health services and delivery research systematic reviews. The prevalence was lower among reviews of associations than among reviews of interventions. The case studies in work package 3 highlighted limitations in current methods for detecting these biases due to heterogeneity and potential confounders. Follow-up of health services and delivery research cohorts in work package 4 showed positive association between publication status and having statistically significant or positive findings. Diverse views concerning publication and related bias and insights into how features of health services and delivery research might influence its occurrence were uncovered through the interviews with health services and delivery research stakeholders and focus group discussion conducted in work package 5.

Conclusions

This study provided prima facie evidence on publication and related bias in quantitative health services and delivery research. This bias does appear to exist, but its prevalence and impact may vary depending on study characteristics, such as study design, and motivation for conducting the evaluation. Emphasis on methodological novelty and focus beyond summative assessments may mitigate/lessen the risk of such bias in health services and delivery research. Methodological and epistemological diversity in health services and delivery research and changing landscape in research publication need to be considered when interpreting the evidence. Collection of further empirical evidence and exploration of optimal health services and delivery research practice are required.

Study registration

This study is registered as PROSPERO CRD42016052333 and CRD42016052366.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 33. See the NIHR Journals Library website for further project information.

Quality and Safety in eHealth: The Need to Build the Evidence Base

Research in the area of health technology safety has demonstrated that technology may both improve patient safety and introduce new types of technology-induced errors. Thus, there is a need to publish safety science literature to develop an evidence-based research base, on which we can continually develop new, safe technologies and improve patient safety. The aim of this viewpoint is to argue for the need to advance evidence-based research in health informatics, so that new technologies can be designed, developed, and implemented for their safety prior to their use in health care. This viewpoint offers a historical perspective on the development of health informatics and safety literature in the area of health technology. I argue for the need to conduct safety studies of technologies used by health professionals and consumers to develop an evidence base in this area. Ongoing research is necessary to improve the quality and safety of health technologies. Over the past several decades, we have seen health informatics emerge as a discipline, with growing research in the field examining the design, development, and implementation of different health technologies and new challenges such as those associated with the quality and safety of technology use. Future research will need to focus on how we can continually extend safety science in this area. There is a need to integrate evidence-based research into the design, development, and implementation of health technologies to improve their safety and reduce technology-induced errors.

Evidence-based Health Informatics: How Do We Know What We Know?

Background: Health IT is expected to have a positive impact on the quality and efficiency of health care. But reports on negative impact and patient harm continue to emerge. The obligation of health informatics is to make sure that health IT solutions provide as much benefit with as few negative side effects as possible. To achieve this, health informatics as a discipline must be able to learn, both from its successes as well as from its failures.

Objectives: To present motivation, vision, and history of evidence-based health informatics, and to discuss achievements, challenges, and needs for action.

Methods: Reflections on scientific literature and on own experiences.

Results: Eight challenges on the way towards evidence-based health informatics are identified and discussed: quality of studies; publication bias; reporting quality; availability of publications; systematic reviews and meta-analysis; training of health IT evaluation experts; translation of evidence into health practice; and post-market surveil-lance. Identified needs for action comprise: establish health IT study registers; increase the quality of publications; develop a taxonomy for health IT systems; improve indexing of published health IT evaluation papers; move from meta-analysis to meta-summaries; include health IT evaluation competencies in curricula; develop evidence-based implementation frameworks; and establish post-marketing surveillance for health IT.

Conclusions: There has been some progress, but evidence-based health informatics is still in its infancy. Building evidence in health informatics is our obligation if we consider medical informatics a scientific discipline.

Discussion of “Evidence-based Health Informatics: How Do We Know What We Know?”

This article is part of a For-Discussion-Section of Methods of Information in Medicine about the paper “Evidence-based Health Informatics: How Do We Know What We Know?” written by Elske Ammenwerth [1]. It is introduced by an editorial. This article contains the combined commentaries invited to independently comment on the Ammenwerth paper. In subsequent issues the discussion can continue through letters to the editor.

With these comments on the paper “Evidence-based Health Informatics: How do we know what we know?”, written by Elske Ammenwerth [1], the journal seeks to stimulate a broad discussion on the challenges of evaluating information processing and information technology in health care. An international group of experts has been invited by the editor of Methods to comment on this paper. Each of the invited commentaries forms one section of this paper.

Impact of IPMOE on nursing tasks in the medical ward: A time-motion study

Introduction

The In-patient Medication Order Entry System (IPMOE) was first implemented in the medical ward of Princess Margaret Hospital, Hong Kong. It was a local developed close-loop system including prescription, dispensing and administration modules. Evaluation on its impact on nursing tasks would be important for practice improvement and subsequent system enhancement.

Objective

The study was conducted to quantify the nursing times across medication-associated tasks for paper-based MAR and computer-based IPMOE, including change in the tasks and time patterns before and after IPMOE implementation.

Methods

This was a prospective observation study in medical wards before (Jan 2014–Jun 2014) and after (Mar 2015–Jun 2015) the implementation of IPMOE. We conducted 8-hr observation studies of individual nurses with a customized application to time various pre-categorized nursing tasks. Statistical inferences and interrupted time series analysis was performed to identify the change in the intercept and trends over time after implementation.

Result

The average number of medication-related tasks was significantly reduced from 61.07 to 29.81, a reduction of 31.26 episodes per duty (P < 0.001, 95% CI 22.9–39.63). The time for the medication-related tasks was reduced from 32 min (SD = 21.57) to 26.57 min (SD = 11.35) and the medication administration time increased from 37.93 min (SD = 14.78) to 44.37 min (SD = 19.45), but there was no overall significant difference in the time spent on each duty (P = 0.315) between the two groups. An improving trend in the delayed effect was observed (P = 0.03), which indicated a run-in period for new application was needed in clinical setting.

Conclusion

Our study had shown the time motion observation could be applied to measure the impact of the IPMOE in a busy clinical setting. Through classification of activities, validation, objective measurement and longitudinal evaluation, the method could be applied in various systems as well as different clinical settings in measure efficiency.

Successful Implementation of Clinical Information Technology: Seven Key Lessons from CPOE

OBJECTIVES

To identify and describe the most critical strategic and operational contributors to the successful implementation of clinical information technologies, as deployed within a moderate sized system of U.S. community hospitals.

BACKGROUND AND SETTING

CHRISTUS Health is a multi-state system comprised of more than 350 services and 60 hospitals with over 9 000 physicians. The Santa Rosa region of CHRISTUS Health, located in greater San Antonio, Texas is comprised of three adult community hospital facilities and one Children's hospital each with bed capacities of 142-180. Computerized Patient Order Entry (CPOE) was first implemented in 2012 within a complex market environment. The Santa Rosa region has 2 417 credentialed physicians and 263 mid-level allied health professionals.

METHODS

This report focuses on the seven most valuable strategies deployed by the Health Informatics team in a large four hospital CHRISTUS region to achieve strong CPOE adoption and critical success lessons learned. The findings are placed within the context of the literature describing best practices in health information technology implementation.

RESULTS

While the elements described involved discrete de novo process generation to support implementation and operations, collectively they represent the creation of a new customer-centric service culture in our Health Informatics team, which has served as a foundation for ensuring strong clinical information technology adoption beyond CPOE.

CONCLUSION

The seven success factors described are not limited in their value to and impact on CPOE adoption, but generalize to - and can advance success in - varied other clinical information technology implementations across diverse hospitals. A number of these factors are supported by reports in the literature of other institutions' successful implementations of CPOE and other clinical information technologies, and while not prescriptive to other settings, may be adapted to yield value elsewhere.

Fostering acceptance of computerized physician order entry: insights from an implementation study

Computerized physician order entry (CPOE) allows physicians to enter orders in a computer rather than handwriting them. Computerized physician order entry is touted as a major improvement in patient safety, and although the literature suggests that such systems have the potential to improve patient outcomes, studies also suggest that CPOE may have significant drawbacks that accompany those benefits. Physicians have often been resistant to accept its implementation. This study investigates the implementation of CPOE at a 217-bed rural hospital in the southeastern United States. Drawing on a mixed-method approach, we identify correlates of change acceptance and propose a set of recommendations for health care managers to foster acceptance of CPOE. Findings from physician surveys (n = 19) indicate that older physicians are less accepting of CPOE, but high-quality change communication may overcome resistance even among older physicians. With insights derived from the organizational change literature, findings bring to the fore a set of practices that managers can use to foster acceptance of CPOE. The thrust of these practices is that managers should make physicians active participants in fine-tuning CPOE within the unique needs and constraints of the local hospital setting.

Clinical decision support systems in the care of inpatients with diabetes in non-critical care setting: systematic review

BACKGROUND

Computerized clinical decision support systems have been claimed to reduce prescription errors and improve patient care. They may play an important role in the care of hospitalized patients with diabetes.

AIM

To collate evidence for the use of clinical decision support systems in improving the care of hospitalized patients with diabetes in a non-critical care setting and to assess their effectiveness.

METHODS

We searched four databases from 1980 to 2010 without language restrictions. All types of studies other than case reports were included. Data extraction and quality assessment were carried out based on the Centre for Review and Dissemination guidance. A narrative synthesis was conducted.

RESULTS

Fourteen studies met the inclusion criteria, including two cluster randomized controlled trials, eight before-and-after studies and four other descriptive studies. Generally, the quality of the studies was not very high. Nine out of 10 studies reported reduction in mean blood glucose or similar measures (patient-day-weighted mean blood glucose) during inpatient stay. The reduction using computerized physician order entry system in patient-day-weighted mean blood glucose ranged from 0.6 to 0.8 mmol/l (10.8-15.6 mg/dl). Other beneficial effects during inpatient stay included reduced use of sliding scale insulin and greater use of basal-bolus insulin regimen. Only one study found a significant increase in hypoglycaemic events.

CONCLUSIONS

Clinical decision support systems have been used, often as part of a complex programme, to improve the care of hospitalized patients with diabetes. There is some evidence that they may have a beneficial effect, but this needs further confirmation.

Reviewing the impact of computerized provider order entry on clinical outcomes: The quality of systematic reviews

PURPOSE

Computerized provider order entry (CPOE) is central to current efforts at improving clinical care. Understanding the quality of the evidence for CPOE is important to the practical decision of implementation, patient safety and future design efforts. This paper presents the results of a systematic analysis of the quality of systematic reviews of empirical CPOE research.

METHODS

The systematic search process included PubMed, CINAHL, Scopus, Cochrane, INSPEC, and PsychInfo databases from the years 1987-mid 2010 in English only. All reviews with a focus on CPOE, electronic ordering, Electronic Health Record, or Health Information Technology were included. Studies were excluded if they did not mention a systematic review in the title or text, report a formal search process, report results of the search, or specifically include a separate section on CPOE in the results. Quality was assessed using systematic criteria developed by Oxman and Guyatt, QUOROM, and PRISMA. All three authors conducted the reviews independently. Disagreements were resolved through discussion. Descriptive data was extracted.

RESULTS

The search process yielded 185 initial unique references with 13 final reviews meeting the inclusion criteria. The rating of overall quality in the Oxman and Guyatt scale averaged 4.9 out of a possible 7 and the average mean of the sum of the other questions was 5.69. The overall QUOROM/PRISMA ratings averaged 63% completion and ranging from 45% to 81%.

CONCLUSIONS

The quality of these reviews were moderate. Only one study conducted a full quantitative synthesis, and overall heterogeneity was reported as very high in the 3 studies that measured it. Recommendations emphasize clarifying the phenomenon of CPOE by avoiding reporting conclusions across sub-group analyses, increasing emphasis on the development of theoretical models, including more quantitative assessments, and increasing breadth of outcomes.

The impact of eHealth on the quality and safety of health care: a systematic overview

BACKGROUND

There is considerable international interest in exploiting the potential of digital solutions to enhance the quality and safety of health care. Implementations of transformative eHealth technologies are underway globally, often at very considerable cost. In order to assess the impact of eHealth solutions on the quality and safety of health care, and to inform policy decisions on eHealth deployments, we undertook a systematic review of systematic reviews assessing the effectiveness and consequences of various eHealth technologies on the quality and safety of care.

METHODS AND FINDINGS

We developed novel search strategies, conceptual maps of health care quality, safety, and eHealth interventions, and then systematically identified, scrutinised, and synthesised the systematic review literature. Major biomedical databases were searched to identify systematic reviews published between 1997 and 2010. Related theoretical, methodological, and technical material was also reviewed. We identified 53 systematic reviews that focused on assessing the impact of eHealth interventions on the quality and/or safety of health care and 55 supplementary systematic reviews providing relevant supportive information. This systematic review literature was found to be generally of substandard quality with regards to methodology, reporting, and utility. We thematically categorised eHealth technologies into three main areas: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. We found that despite support from policymakers, there was relatively little empirical evidence to substantiate many of the claims made in relation to these technologies. Whether the success of those relatively few solutions identified to improve quality and safety would continue if these were deployed beyond the contexts in which they were originally developed, has yet to be established. Importantly, best practice guidelines in effective development and deployment strategies are lacking.

CONCLUSIONS

There is a large gap between the postulated and empirically demonstrated benefits of eHealth technologies. In addition, there is a lack of robust research on the risks of implementing these technologies and their cost-effectiveness has yet to be demonstrated, despite being frequently promoted by policymakers and "techno-enthusiasts" as if this was a given. In the light of the paucity of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, it is vital that future eHealth technologies are evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle. Such evaluation should be characterised by careful attention to socio-technical factors to maximise the likelihood of successful implementation and adoption.

The effect of electronic prescribing on medication errors and adverse drug events: a systematic review

The objective of this systematic review is to analyse the relative risk reduction on medication error and adverse drug events (ADE) by computerized physician order entry systems (CPOE). We included controlled field studies and pretest-posttest studies, evaluating all types of CPOE systems, drugs and clinical settings. We present the results in evidence tables, calculate the risk ratio with 95% confidence interval and perform subgroup analyses for categorical factors, such as the level of care, patient group, type of drug, type of system, functionality of the system, comparison group type, study design, and the method for detecting errors. Of the 25 studies that analysed the effects on the medication error rate, 23 showed a significant relative risk reduction of 13% to 99%. Six of the nine studies that analysed the effects on potential ADEs showed a significant relative risk reduction of 35% to 98%. Four of the seven studies that analysed the effect on ADEs showed a significant relative risk reduction of 30% to 84%. Reporting quality and study quality was often insufficient to exclude major sources of bias. Studies on home-grown systems, studies comparing electronic prescribing to handwriting prescribing, and studies using manual chart review to detect errors seem to show a higher relative risk reduction than other studies. Concluding, it seems that electronic prescribing can reduce the risk for medication errors and ADE. However, studies differ substantially in their setting, design, quality, and results. To further improve the evidence-base of health informatics, more randomized controlled trials (RCTs) are needed, especially to cover a wider range of clinical and geographic settings. In addition, reporting quality of health informatics evaluation studies has to be substantially improved.