Developing and maintaining clinical decision support using clinical knowledge and machine learning: the case of order sets

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.

Machine learning models for synthesizing actionable care decisions on lower extremity wounds

Lower extremity chronic wounds affect 4.5 million Americans annually. Due to inadequate access to wound experts in underserved areas, many patients receive non-uniform, non-standard wound care, resulting in increased costs and lower quality of life. We explored machine learning classifiers to generate actionable wound care decisions about four chronic wound types (diabetic foot, pressure, venous, and arterial ulcers). These decisions (target classes) were: (1) Continue current treatment, (2) Request non-urgent change in treatment from a wound specialist, (3) Refer patient to a wound specialist. We compare classification methods (single classifiers, bagged & boosted ensembles, and a deep learning network) to investigate (1) whether visual wound features are sufficient for generating a decision and (2) whether adding unstructured text from wound experts increases classifier accuracy. Using 205 wound images, the Gradient Boosted Machine (XGBoost) outperformed other methods when using both visual and textual wound features, achieving 81% accuracy. Using only visual features decreased the accuracy to 76%, achieved by a Support Vector Machine classifier. We conclude that machine learning classifiers can generate accurate wound care decisions on lower extremity chronic wounds, an important step toward objective, standardized wound care. Higher decision-making accuracy was achieved by leveraging clinical comments from wound experts.

ClinicNet: machine learning for personalized clinical order set recommendations

OBJECTIVE

This study assesses whether neural networks trained on electronic health record (EHR) data can anticipate what individual clinical orders and existing institutional order set templates clinicians will use more accurately than existing decision support tools.

MATERIALS AND METHODS

We process 57 624 patients worth of clinical event EHR data from 2008 to 2014. We train a feed-forward neural network (ClinicNet) and logistic regression applied to the traditional problem structure of predicting individual clinical items as well as our proposed workflow of predicting existing institutional order set template usage.

RESULTS

ClinicNet predicts individual clinical orders (precision = 0.32, recall = 0.47) better than existing institutional order sets (precision = 0.15, recall = 0.46). The ClinicNet model predicts clinician usage of existing institutional order sets (avg. precision = 0.31) with higher average precision than a baseline of order set usage frequencies (avg. precision = 0.20) or a logistic regression model (avg. precision = 0.12).

DISCUSSION

Machine learning methods can predict clinical decision-making patterns with greater accuracy and less manual effort than existing static order set templates. This can streamline existing clinical workflows, but may not fit if historical clinical ordering practices are incorrect. For this reason, manually authored content such as order set templates remain valuable for the purposeful design of care pathways. ClinicNet's capability of predicting such personalized order set templates illustrates the potential of combining both top-down and bottom-up approaches to delivering clinical decision support content.

CONCLUSION

ClinicNet illustrates the capability for machine learning methods applied to the EHR to anticipate both individual clinical orders and existing order set templates, which has the potential to improve upon current standards of practice in clinical order entry.

Artificial Intelligence Pertaining to Cardiothoracic Imaging and Patient Care: Beyond Image Interpretation

Artificial intelligence (AI) is a broad field of computational science that includes many subsets. Today the most widely used subset in medical imaging is machine learning (ML). Many articles have focused on the use of ML for pattern recognition to detect and potentially diagnose various pathologies. However, AI algorithm development is now directed toward workflow management. AI can impact patient care at multiple stages of their imaging experience and assist in efficient and effective scheduling, imaging performance, worklist prioritization, image interpretation, and quality assurance. The purpose of this manuscript was to review the potential AI applications in radiology focusing on workflow management and discuss how ML will affect cardiothoracic imaging.

When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record

BACKGROUND

Order sets are widely used tools in the electronic health record (EHR) for improving healthcare quality. However, there is limited insight into how well they facilitate clinician workflow. We assessed four indicators based on order set usage patterns in the EHR that reflect potential misalignment between order set design and clinician workflow needs.

METHODS

We used data from the EHR on all orders of medication, laboratory, imaging and blood product items at an academic hospital and an itemset mining approach to extract orders that frequently co-occurred with order set use. We identified the following four indicators: infrequent ordering of order set items, rapid retraction of medication orders from order sets, additional a la carte ordering of items not included in order sets and a la carte ordering of items despite being listed in the order set.

RESULTS

There was significant variability in workflow alignment across the 11 762 order set items used in the 77 421 inpatient encounters from 2014 to 2017. The median ordering rate was 4.1% (IQR 0.6%-18%) and median medication retraction rate was 4% (IQR 2%-10%). 143 (5%) medications were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte. 214 (39%) order sets were associated with least one additional item frequently ordered a la carte and 243 (45%) order sets contained at least one item that was instead more often ordered a la carte.

CONCLUSION

Order sets often do not align with what clinicians need at the point of care. Quantitative insights from EHRs may inform how order sets can be optimised to facilitate clinician workflow.

Developing data-driven clinical pathways using electronic health records: The cases of total laparoscopic hysterectomy and rotator cuff tears

OBJECTIVE

A clinical pathway is one of the tools used to support clinical decision making that provides a standardized care process in a specific context. The objective of this research was to develop a method for building data-driven clinical pathways using electronic health record data.

MATERIALS AND METHODS

We proposed a matching rate-based clinical pathway mining algorithm that produces the optimal set of clinical orders for each clinical stage by employing matching rates. To validate the approach, we utilized two different datasets of deidentified inpatient records directly related to total laparoscopic hysterectomy (TLH) and rotator cuff tears (RCTs) from a hospital in South Korea. The derived data-driven clinical pathways were evaluated with knowledge-based models by health professionals using a delta analysis.

RESULTS

Two different data-driven clinical pathways, i.e., TLH and RCTs, were produced by applying the matching rate-based clinical pathway mining algorithm. We identified that there were significant differences in clinical orders between the data-driven and knowledge-based models. Additionally, the data-driven clinical pathways based on our algorithm outperformed the models by clinical experts, with average matching rates of 82.02% and 79.66%, respectively.

CONCLUSION

The proposed algorithm will be helpful for supporting clinical decisions and directly applicable in medical practices.