Think time: A novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts

Optimising interruptive clinical decision support alerts for antithrombotic duplicate prescribing in hospital

INTRODUCTION

Duplicate prescribing clinical decision support alerts can prevent important prescribing errors but are frequently the cause of much alert fatigue. Stat dose prescriptions are a known reason for overriding these alerts. This study aimed to evaluate the effect of excluding stat dose prescriptions from duplicate prescribing alerts for antithrombotic medicines on alert burden, prescriber adherence, and prescribing.

MATERIALS AND METHODS

A before (January 1st, 2017 to August 31st, 2022) and after (October 5th, 2022 to September 30th, 2023) study was undertaken of antithrombotic duplicate prescribing alerts and prescribing following a change in alert settings. Alert and prescribing data for antithrombotic medicines were joined, processed, and analysed to compare alert rates, adherence, and prescribing. Alert burden was assessed as alerts per 100 prescriptions. Adherence was measured at the point of the alert as whether the prescriber accepted the alert and following the alert as whether a relevant prescription was ceased within an hour. Co-prescribing of antithrombotic stat dose prescriptions was assessed pre- and post-alert reconfiguration.

RESULTS

Reconfiguration of the alerts reduced the alert rate by 29 % (p < 0.001). The proportion of alerts associated with cessation of antithrombotic duplication significantly increased (32.8 % to 44.5 %, p < 0.001). Adherence at the point of the alert increased 1.2 % (4.8 % to 6.0 %, p = 0.012) and 11.5 % (29.4 % to 40.9 %, p < 0.001) within one hour of the alert. When ceased after the alert over 80 % of duplicate prescriptions were ceased within 2 min of overriding. Antithrombotic stat dose co-prescribing was unchanged for 4 out of 5 antithrombotic duplication alert rules.

CONCLUSION

By reconfiguring our antithrombotic duplicate prescribing alerts, we reduced alert burden and increased alert adherence. Many prescribers ceased duplicate prescribing within 2 min of alert override highlighting the importance of incorporating post-alert measures in accurately determining prescriber alert adherence.

Using alert dwell time to filter universal clinical alerts: A machine learning approach

BACKGROUND

Alerts in computerized physician order entry (CPOE) systems can improve patient safety. However, alerts in rule-based systems cannot be customized based on individual patient or user characteristics. This limitation can lead to the presentation of irrelevant alerts and subsequent alert fatigue.

OBJECTIVE

We used machine learning approaches with alert dwell time to filter out irrelevant alerts for physicians based on contextual factors.

METHODS

We utilized five machine learning algorithms and a total of 1,120 features grouped into six categories: alert, demographic, environment, diagnosis, prescription, and laboratory results. The output of the models was the alert dwell time within a specified time window to determine the optimal range by the sensitivity analysis.

RESULTS

We used 813,026 records (19 categories) from the hospital's outpatient clinic data from 2020 to 2021. The sensitivity analysis showed that a time window with a range of 0.3-4.0 s had the best performance, with an area under the receiver operating characteristic (AUROC) curve of 0.73 and an area under the precision-recall curve (AUPRC) of 0.97. The model built with alert and demographic feature groups showed the best performance, with an AUROC of 0.73. The most significant individual feature groups were alert and demographic, with AUROCs of 0.66 and 0.62, respectively.

CONCLUSION

Our study found that alerts and user and patient demographic features are more crucial than clinical features when constructing universal context-aware alerts. Using alert dwell time in combination with a time window is an effective way to determine the trigger status of an alert. The findings of this study can provide useful insights for researchers working on specific and universal context-aware alerts.

Optimizing Best Practice Advisory alerts in electronic medical records with a multi-pronged strategy at a tertiary care hospital in Singapore

Objective

Clinical decision support (CDS) alerts can aid in improving patient care. One CDS functionality is the Best Practice Advisory (BPA) alert notification system, wherein BPA alerts are automated alerts embedded in the hospital's electronic medical records (EMR). However, excessive alerts can change clinician behavior; redundant and repetitive alerts can contribute to alert fatigue. Alerts can be optimized through a multipronged strategy. Our study aims to describe these strategies adopted and evaluate the resultant BPA alert optimization outcomes.

Materials and Methods

This retrospective single-center study was done at Jurong Health Campus. Aggregated, anonymized data on patient demographics and alert statistics were collected from January 1, 2018 to December 31, 2021. "Preintervention" period was January 1-December 31, 2018, and "postintervention" period was January 1-December 31, 2021. The intervention period was the intervening period. Categorical variables were reported as frequencies and proportions and compared using the chi-square test. Continuous data were reported as median (interquartile range, IQR) and compared using the Wilcoxon rank-sum test. Statistical significance was defined at P < .05.

Results

There was a significant reduction of 59.6% in the total number of interruptive BPA alerts, despite an increase in the number of unique BPAs from 54 to 360 from pre- to postintervention. There was a 74% reduction in the number of alerts from the 7 BPAs that were optimized from the pre- to postintervention period. There was a significant increase in percentage of overall interruptive BPA alerts with action taken (8 [IQR 7.7-8.4] to 54.7 [IQR 52.5-58.9], P-value < .05) and optimized BPAs with action taken (32.6 [IQR 32.3-32.9] to 72.6 [IQR 64.3-73.4], P-value < .05). We estimate that the reduction in alerts saved 3600 h of providers' time per year.

Conclusions

A significant reduction in interruptive alert volume, and a significant increase in action taken rates despite manifold increase in the number of unique BPAs could be achieved through concentrated efforts focusing on governance, data review, and visualization using a system-embedded tool, combined with the CDS Five Rights framework, to optimize alerts. Improved alert compliance was likely multifactorial-due to decreased repeated alert firing for the same patient; better awareness due to stakeholders' involvement; and less fatigue since unnecessary alerts were removed. Future studies should prospectively focus on patients' clinical chart reviews to assess downstream effects of various actions taken, identify any possibility of harm, and collect end-user feedback regarding the utility of alerts.

Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework

Alerts are a common functionality of clinical decision support systems (CDSSs). Although they have proven to be useful in clinical practice, the alert burden can lead to alert fatigue and significantly reduce their usability and acceptance. Based on a literature review, we propose a unified framework consisting of a set of meaningful timestamps that allows the use of state-of-the-art measures for alert burden, such as alert dwell time, alert think time, and response time. In addition, it can be used to investigate other measures that could be relevant as regards dealing with this problem. Furthermore, we provide a case study concerning three different types of alerts to which the framework was successfully applied. We consider that our framework can easily be adapted to other CDSSs and that it could be useful for dealing with alert burden measurement thus contributing to its appropriate management.

Drug-Drug-Gene Interactions in Cardiovascular Medicine

Cardiovascular disease remains a leading cause of both morbidity and mortality worldwide. It is widely accepted that both concomitant medications (drug-drug interactions, DDIs) and genomic factors (drug-gene interactions, DGIs) can influence cardiovascular drug-related efficacy and safety outcomes. Although thousands of DDI and DGI (aka pharmacogenomic) studies have been published to date, the literature on drug-drug-gene interactions (DDGIs, cumulative effects of DDIs and DGIs) remains scarce. Moreover, multimorbidity is common in cardiovascular disease patients and is often associated with polypharmacy, which increases the likelihood of clinically relevant drug-related interactions. These, in turn, can lead to reduced drug efficacy, medication-related harm (adverse drug reactions, longer hospitalizations, mortality) and increased healthcare costs. To examine the extent to which DDGIs and other interactions influence efficacy and safety outcomes in the field of cardiovascular medicine, we review current evidence in the field. We describe the different categories of DDIs and DGIs before illustrating how these two interact to produce DDGIs and other complex interactions. We provide examples of studies that have reported the prevalence of clinically relevant interactions and the most implicated cardiovascular medicines before outlining the challenges associated with dealing with these interactions in clinical practice. Finally, we provide recommendations on how to manage the challenges including but not limited to expanding the scope of drug information compendia, interaction databases and clinical implementation guidelines (to include clinically relevant DDGIs and other complex interactions) and work towards their harmonization; better use of electronic decision support tools; using big data and novel computational techniques; using clinically relevant endpoints, preemptive genotyping; ensuring ethnic diversity; and upskilling of clinicians in pharmacogenomics and personalized medicine.

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.

Modulators Influencing Medication Alert Acceptance: An Explorative Review

OBJECTIVES

Clinical decision support systems (CDSSs) use alerts to enhance medication safety and reduce medication error rates. A major challenge of medication alerts is their low acceptance rate, limiting their potential benefit. A structured overview about modulators influencing alert acceptance is lacking. Therefore, we aimed to review and compile qualitative and quantitative modulators of alert acceptance and organize them in a comprehensive model.

METHODS

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline, a literature search in PubMed was started in February 2018 and continued until October 2021. From all included articles, qualitative and quantitative parameters and their impact on alert acceptance were extracted. Related parameters were then grouped into factors, allocated to superordinate determinants, and subsequently further allocated into five categories that were already known to influence alert acceptance.

RESULTS

Out of 539 articles, 60 were included. A total of 391 single parameters were extracted (e.g., patients' comorbidity) and grouped into 75 factors (e.g., comorbidity), and 25 determinants (e.g., complexity) were consequently assigned to the predefined five categories, i.e., CDSS, care provider, patient, setting, and involved drug. More than half of all factors were qualitatively assessed (n = 21) or quantitatively inconclusive (n = 19). Furthermore, 33 quantitative factors clearly influenced alert acceptance (positive correlation: e.g., alert type, patients' comorbidity; negative correlation: e.g., number of alerts per care provider, moment of alert display in the workflow). Two factors (alert frequency, laboratory value) showed contradictory effects, meaning that acceptance was significantly influenced both positively and negatively by these factors, depending on the study. Interventional studies have been performed for only 12 factors while all other factors were evaluated descriptively.

CONCLUSION

This review compiles modulators of alert acceptance distinguished by being studied quantitatively or qualitatively and indicates their effect magnitude whenever possible. Additionally, it describes how further research should be designed to comprehensively quantify the effect of alert modulators.

dfgcompare: a library to support process variant analysis through Markov models

BACKGROUND

Data-driven process analysis is an important area that relies on software support. Process variant analysis is a sort of analysis technique in which analysts compare executed process variants, a.k.a. process cohorts. This comparison can help to identify insights for improving processes. There are a few software supports to enable process cohort comparison based on the frequencies of process activities and performance metrics. These metrics are effective in cohort analysis, but they cannot support cohort comparison based on the probability of transitions among states, which is an important enabler for cohort analysis in healthcare.

RESULTS

This paper defines an approach to compare process cohorts using Markov models. The approach is formalized, and it is implemented as an open-source python library, named dfgcompare. This library can be used by other researchers to compare process cohorts. The implementation is also used to compare caregivers' behavior when prescribing drugs in the Stockholm Region. The result shows that the approach enables the comparison of process cohorts in practice.

CONCLUSIONS

We conclude that dfgcompare supports identifying differences among process cohorts.

Measuring the impact of cardiotocograph decision support software on neonatal outcomes: A propensity score-matched observational study

BACKGROUND

This study follows the 2017 UK INFANT Collaborative Group RCT, which compared neonatal outcomes with and without the use of the INFANT cardiotocograph decision support system for over 46 000 patients in labour. The original trial failed to demonstrate a significant improvement to neonatal outcomes; however, the study design was subject to methodological critique.

AIMS

This Australian retrospective cohort study aimed to report perinatal outcomes before and after the introduction of INFANT decision support software for cardiotocograph use in labour.

MATERIALS AND METHODS

The study cohort was divided into two equivalent 18-month epochs, before and after the introduction of INFANT-Guardian® CTG decision support system. Propensity score matching analysis was undertaken to balance pre- and post-implementation groups by baseline covariates. The matched cohort included 11 154 public-funded women between November 2016 and 2019, with a singleton live fetus ≥34 + 0 weeks, being induced or in spontaneous labour. The main outcome measures were: a composite measure of serious adverse neonatal outcome comprising of one or more of: admission to intensive care nursery >48 h, Apgar <4 at 5 min, cord arterial pH <7.0, hypoxic ischaemic encephalopathy grade 2 or 3, therapeutic hypothermia, neonatal death.

RESULTS

The incidence of the composite primary outcome was significantly lower following implementation of INFANT (0.57% vs. 1.00%; OR 0.57, 95%CI 0.37-0.88; P = 0.01). A significant reduction in nursery admission >48 h was also observed (0.05% vs. 0.30%; OR 0.18, 95%CI 0.05-0.60; P = 0.002).

CONCLUSIONS

INFANT software is associated with a reduction in serious adverse neonatal outcomes, without increasing the rate of operative delivery.

Reducing Alert Burden in Electronic Health Records: State of the Art Recommendations from Four Health Systems

BACKGROUND

Electronic health record (EHR) alert fatigue, while widely recognized as a concern nationally, lacks a corresponding comprehensive mitigation plan.

OBJECTIVES

The goal of this manuscript is to provide practical guidance to clinical informaticists and other health care leaders who are considering creating a program to manage EHR alerts.

METHODS

This manuscript synthesizes several approaches and recommendations for better alert management derived from four U.S. health care institutions that presented their experiences and recommendations at the American Medical Informatics Association 2019 Clinical Informatics Conference in Atlanta, Georgia, United States. The assembled health care institution leaders represent academic, pediatric, community, and specialized care domains. We describe governance and management, structural concepts and components, and human-computer interactions with alerts, and make recommendations regarding these domains based on our experience supplemented with literature review. This paper focuses on alerts that impact bedside clinicians.

RESULTS

The manuscript addresses the range of considerations relevant to alert management including a summary of the background literature about alerts, alert governance, alert metrics, starting an alert management program, approaches to evaluating alerts prior to deployment, and optimization of existing alerts. The manuscript includes examples of alert optimization successes at two of the represented institutions. In addition, we review limitations on the ability to evaluate alerts in the current state and identify opportunities for further scholarship.

CONCLUSION

Ultimately, alert management programs must strive to meet common goals of improving patient care, while at the same time decreasing the alert burden on clinicians. In so doing, organizations have an opportunity to promote the wellness of patients, clinicians, and EHRs themselves.

The Effect of Eliminating Intermediate Severity Drug-Drug Interaction Alerts on Overall Medication Alert Burden and Acceptance Rate

OBJECTIVE

This study aimed to determine the effects of reducing the number of drug-drug interaction (DDI) alerts in an order entry system.

METHODS

Retrospective pre-post analysis at an urban medical center of the rates of medication alerts and alert acceptance during a 5-month period before and 5-month period after the threshold for firing DDI alerts was changed from "intermediate" to "severe." To ensure that we could determine varying response to each alert type, we took an in-depth look at orders generating single alerts.

RESULTS

Before the intervention, 241,915 medication orders were placed, of which 25.6% generated one or more medication alerts; 5.3% of the alerts were accepted. During the postintervention period, 245,757 medication orders were placed of which 16.0% generated one or more medication alerts, a 37.5% relative decrease in alert rate (95% confidence interval [CI]: -38.4 to -36.8%), but only a 9.6% absolute decrease (95% CI: -9.4 to -9.9%). 7.4% of orders generating alerts were accepted postintervention, a 39.6% relative increase in acceptance rate (95% CI: 33.2-47.2%), but only a 2.1% absolute increase (95% CI: 1.8-2.4%). When only orders generating a single medication alert were considered, there was a 69.1% relative decrease in the number of orders generating DDI alerts, and an 85.7% relative increase in the acceptance rate (95% CI: 58.6-126.2%), though only a 1.8% absolute increase (95% CI: 1.3-2.3%).

CONCLUSION

Eliminating intermediate severity DDI alerts resulted in a statistically significant decrease in alert burden and increase in the rate of medication alert acceptance, but alert acceptance remained low overall.

Medication-related clinical decision support alert overrides in inpatients

Objective

To define the types and numbers of inpatient clinical decision support alerts, measure the frequency with which they are overridden, and describe providers' reasons for overriding them and the appropriateness of those reasons.

Materials and Methods

We conducted a cross-sectional study of medication-related clinical decision support alerts over a 3-year period at a 793-bed tertiary-care teaching institution. We measured the rate of alert overrides, the rate of overrides by alert type, the reasons cited for overrides, and the appropriateness of those reasons.

Results

Overall, 73.3% of patient allergy, drug-drug interaction, and duplicate drug alerts were overridden, though the rate of overrides varied by alert type (P < .0001). About 60% of overrides were appropriate, and that proportion also varied by alert type (P < .0001). Few overrides of renal- (2.2%) or age-based (26.4%) medication substitutions were appropriate, while most duplicate drug (98%), patient allergy (96.5%), and formulary substitution (82.5%) alerts were appropriate.

Discussion

Despite warnings of potential significant harm, certain categories of alert overrides were inappropriate >75% of the time. The vast majority of duplicate drug, patient allergy, and formulary substitution alerts were appropriate, suggesting that these categories of alerts might be good targets for refinement to reduce alert fatigue.

Conclusion

Almost three-quarters of alerts were overridden, and 40% of the overrides were not appropriate. Future research should optimize alert types and frequencies to increase their clinical relevance, reducing alert fatigue so that important alerts are not inappropriately overridden.

Adverse drug reactions caused by drug-drug interactions in cardiovascular disease patients: introduction of a simple prediction tool using electronic screening database items

Objective: Cardiovascular disease (CVD) drugs have been frequently implicated in adverse drug reaction (ADR)-related hospitalizations. Drug-drug interactions (DDIs) are common preventable cause of ADRs, but the impact of DDIs in the CVD population has not been investigated. Hence, the primary aim of the study was to identify DDIs associated with ADRs in CVD patients at hospital admission. The second aim was to develop a simple tool to identify high-risk patients for DDI-related adverse events. Methods: An observational study was conducted on the Cardiology Ward of University Clinical Hospital Center. Data were obtained from medical charts. A clinical panel identified DDIs implicated in ADRs, using LexiInteract database and Drug Interaction Probability Scale. Statistics were performed using PASW 22 (SPSS Inc.). Results: DDIs contributed to hospital admission with a total prevalence of 9.69%. DDI-related ADRs affected mainly cardiac function (heart rate or rhythm, 41.07%); bleeding and effect on blood pressure were equally distributed (17.86%). Non-cardiovascular ADRs were found in 23.21% of DDIs. After admission, 73% of the identified DDIs led to changes in prescription. Prediction ability of calculated DDI adverse event probability scores was rated as good (AUC = 0.80, p < .001). Conclusions: CVD patients are highly exposed to adverse DDIs; about one in ten patients hospitalized with CVD might have a DDI contributing to the hospitalization. Given the high prevalence of CVD, DDI-related harm might be a significant burden worldwide. Identification of patients with high DDI adverse event risk might ease the recognition of DDI-related harm and improve the use of electronic databases in clinical practice.

Optimizing Drug-Drug Interaction Alerts Using a Multidimensional Approach

OBJECTIVES

Excessive alerts are a common concern associated with clinical decision support systems that monitor drug-drug interactions (DDIs). To reduce the number of low-value interruptive DDI alerts at our hospital, we implemented an iterative, multidimensional quality improvement effort, which included an interdisciplinary advisory group, alert metrics, and measurement of perceived clinical value.

METHODS

Alert data analysis indicated that DDIs were the most common interruptive medication alert. An interdisciplinary alert advisory group was formed to provide expert advice and oversight for alert refinement and ongoing review of alert data. Alert data were categorized into drug classes and analyzed to identify DDI alerts for refinement. Refinement strategies included alert suppression and modification of alerts to be contextually aware.

RESULTS

On the basis of historical analysis of classified DDI alerts, 26 alert refinements were implemented, representing 47% of all alerts. Alert refinement efforts resulted in the following substantial decreases in the number of interruptive DDI alerts: 40% for all clinicians (22.9-14 per 100 orders) and as high as 82% for attending physicians (6.5-1.2 per 100 orders). Two patient safety events related to alert refinements were reported during the project period.

CONCLUSIONS

Our quality improvement effort refined 47% of all DDI alerts that were firing during historical analysis, significantly reduced the number of DDI alerts in a 54-week period, and established a model for sustained alert refinements.

Behavioral Economics Interventions in Clinical Decision Support Systems

Background:  Clinical decision support (CDS) systems can improve safety and facilitate evidence-based practice. However, clinical decisions are often affected by the cognitive biases and heuristics of clinicians, which is increasing the interest in behavioral and cognitive science approaches in the medical field.

Objectives:  This review aimed to identify decision biases that lead clinicians to exhibit irrational behaviors or responses, and to show how behavioral economics can be applied to interventions in order to promote and reveal the contributions of CDS to improving health care quality.

Methods:  We performed a systematic review of studies published in 2016 and 2017 and applied a snowball citationsearch method to identify topical publications related to studies forming part of the BEARI (Application of Behavioral Economics to Improve the Treatment of Acute Respiratory Infections) multisite, cluster-randomized controlled trial performed in the United States.

Results:  We found that 10 behavioral economics concepts with nine cognitive biases were addressed and investigated for clinician decision-making, and that the following five concepts, which were actively explored, had an impact in CDS applications: social norms, framing effect, status-quo bias, heuristics, and overconfidence bias.

Conclusions:  Our review revealed that the use of behavioral economics techniques is increasing in areas such as antibiotics prescribing and preventive care, and that additional tests of the concepts and heuristics described would be useful in other areas of CDS. An improved understanding of the benefits and limitations of behavioral economics techniques is also still needed. Future studies should focus on successful design strategies and how to combine them with CDS functions for motivating clinicians.

Comparison of Overridden Medication-related Clinical Decision Support in the Intensive Care Unit between a Commercial System and a Legacy System

BACKGROUND

Electronic health records (EHRs) with clinical decision support (CDS) have shown to be effective at improving patient safety. Despite this, alerts delivered as part of CDS are overridden frequently, which is of concern in the critical care population as this group may have an increased risk of harm. Our organization recently transitioned from an internally-developed EHR to a commercial system. Data comparing various EHR systems, especially after transitions between EHRs, are needed to identify areas for improvement.

OBJECTIVES

To compare the two systems and identify areas for potential improvement with the new commercial system at a single institution.

METHODS

Overridden medication-related CDS alerts were included from October to December of the systems' respective years (legacy, 2011; commercial, 2015), restricted to three intensive care units. The two systems were compared with regards to CDS presentation and override rates for four types of CDS: drug-allergy, drug-drug interaction (DDI), geriatric and renal alerts. A post hoc analysis to evaluate for adverse drug events (ADEs) potentially resulting from overridden alerts was performed for 'contraindicated' DDIs via chart review.

RESULTS

There was a significant increase in provider exposure to alerts and alert overrides in the commercial system (commercial: n=5,535; legacy: n=1,030). Rates of overrides were higher for the allergy and DDI alerts (p<0.001) in the commercial system. Geriatric and renal alerts were significantly different in incidence and presentation between the two systems. No ADEs were identified in an analysis of 43 overridden contraindicated DDI alerts.

CONCLUSIONS

The vendor system had much higher rates of both alerts and overrides, although we did not find evidence of harm in a review of DDIs which were overridden. We propose recommendations for improving our current system which may be helpful to other similar institutions; improving both alert presentation and the underlying knowledge base appear important.

Advances in Clinical Decision Support: Highlights of Practice and the Literature 2015-2016

Introduction: Advances in clinical decision support (CDS) continue to evolve to support the goals of clinicians, policymakers, patients and professional organizations to improve clinical practice, patient safety, and the quality of care. Objectives: Identify key thematic areas or foci in research and practice involving clinical decision support during the 2015-2016 time period. Methods: Thematic analysis consistent with a grounded theory approach was applied in a targeted review of journal publications, the proceedings of key scientific conferences as well as activities in standards development organizations in order to identify the key themes underlying work related to CDS. Results: Ten key thematic areas were identified, including: 1) an emphasis on knowledge representation, with a focus on clinical practice guidelines; 2) various aspects of precision medicine, including the use of sensor and genomic data as well as big data; 3) efforts in quality improvement; 4) innovative uses of computer-based provider order entry (CPOE) systems, including relevant data displays; 5) expansion of CDS in various clinical settings; 6) patient-directed CDS; 7) understanding the potential negative impact of CDS; 8) obtaining structured data to drive CDS interventions; 9) the use of diagnostic decision support; and 10) the development and use of standards for CDS. Conclusions: Active research and practice in 2015-2016 continue to underscore the importance and broad utility of CDS for effecting change and improving the quality and outcome of clinical care.