Screening for medication errors using an outlier detection system

A Machine Learning Application to Classify Patients at Differing Levels of Risk of Opioid Use Disorder: Clinician-Based Validation Study

Background

Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment.

Objective

This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians.

Methods

The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient's OUD risk level, which was then compared to the ML application's risk assignments.

Results

A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified.

Conclusions

A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients' different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.

The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review

OBJECTIVE

Current Clinical Decision Support Systems (CDSSs) generate medication alerts that are of limited clinical value, causing alert fatigue. Artificial Intelligence (AI)-based methods may help in optimizing medication alerts. Therefore, we conducted a scoping review on the current state of the use of AI to optimize medication alerts in a hospital setting. Specifically, we aimed to identify the applied AI methods used together with their performance measures and main outcome measures.

MATERIALS AND METHODS

We searched Medline, Embase, and Cochrane Library database on May 25, 2023 for studies of any quantitative design, in which the use of AI-based methods was investigated to optimize medication alerts generated by CDSSs in a hospital setting. The screening process was supported by ASReview software.

RESULTS

Out of 5625 citations screened for eligibility, 10 studies were included. Three studies (30%) reported on both statistical performance and clinical outcomes. The most often reported performance measure was positive predictive value ranging from 9% to 100%. Regarding main outcome measures, alerts optimized using AI-based methods resulted in a decreased alert burden, increased identification of inappropriate or atypical prescriptions, and enabled prediction of user responses. In only 2 studies the AI-based alerts were implemented in hospital practice, and none of the studies conducted external validation.

DISCUSSION AND CONCLUSION

AI-based methods can be used to optimize medication alerts in a hospital setting. However, reporting on models' development and validation should be improved, and external validation and implementation in hospital practice should be encouraged.

Current perspectives on the use of artificial intelligence in critical patient safety

Intensive Care Units (ICUs) have undergone enhancements in patient safety, and artificial intelligence (AI) emerges as a disruptive technology offering novel opportunities. While the published evidence is limited and presents methodological issues, certain areas show promise, such as decision support systems, detection of adverse events, and prescription error identification. The application of AI in safety may pursue predictive or diagnostic objectives. Implementing AI-based systems necessitates procedures to ensure secure assistance, addressing challenges including trust in such systems, biases, data quality, scalability, and ethical and confidentiality considerations. The development and application of AI demand thorough testing, encompassing retrospective data assessments, real-time validation with prospective cohorts, and efficacy demonstration in clinical trials. Algorithmic transparency and explainability are essential, with active involvement of clinical professionals being crucial in the implementation process.

Development and implementation of medication-related clinical rules for obstetrics, gynaecology, and paediatric outpatients

OBJECTIVES

Prescription errors can cause serious adverse drug events. Clinical decision support systems prevent prescription errors; however, real-time clinical rules in obstetrics, gynaecology, and paediatric outpatients remain unexplored. We evaluated the effects of localised, real-time clinical rules on alert rates and acceptance rates compared with manual prescription review.

METHODS

We developed real-time clinical rules that incorporate information systems to obtain characteristic information and laboratory values. We conducted a retrospective cohort study to compare the alert and recommendation acceptance rates of all prescription error types before and after clinical rule implementation in obstetrics, gynaecology, and paediatrics. Clinical rules, prescription error types, and alerts were determined by a prescribing review committee comprising physicians, pharmacists, nurses, and administrators. The difference in alert and acceptance rates between the groups was analysed using relative risk.

RESULTS

The number of alerts increased after clinical rules implementation; the number of on-duty pharmacists for review decreased from 10 to 2. Compared with those with manual review, the alert rates for paediatrics and obstetrics and gynaecology increased with the clinical rules by 3.97- and 11.26-fold, respectively, and the alert rates for drug-drug interactions (DDIs) and combined medication errors in obstetrics and gynaecology increased with the clinical rules by 26.10- and 26.54-fold, respectively. In paediatrics, the alert rate for all prescription error types was higher with the clinical rules review than with the manual review; the alert rates for DDI, dosage, and combination medication errors were significantly different between the clinical rules and the manual review. However, there was no difference in the recommendation acceptance rate between the manual review and the clinical rules.

CONCLUSIONS

Clinical rules can identify prescription errors that manual review cannot detect and ensure real-time review efficiency in high-volume outpatient prescription settings. The high acceptance rate and modification of prescriptions may be relevant to highly customised and localised clinical rules.

Identifying Anomalies while Preserving Privacy

Identifying anomalies in data is vital in many domains, including medicine, finance, and national security. However, privacy concerns pose a significant roadblock to carrying out such an analysis. Since existing privacy definitions do not allow good accuracy when doing outlier analysis, the notion of sensitive privacy has been recently proposed to deal with this problem. Sensitive privacy makes it possible to analyze data for anomalies with practically meaningful accuracy while providing a strong guarantee similar to differential privacy, which is the prevalent privacy standard today. In this work, we relate sensitive privacy to other important notions of data privacy so that one can port the technical developments and private mechanism constructions from these related concepts to sensitive privacy. Sensitive privacy critically depends on the underlying anomaly model. We develop a novel n-step lookahead mechanism to efficiently answer arbitrary outlier queries, which provably guarantees sensitive privacy if we restrict our attention to common a class of anomaly models. We also provide general constructions to give sensitively private mechanisms for identifying anomalies and show the conditions under which the constructions would be optimal.

Development of artificial intelligence powered apps and tools for clinical pharmacy services: A systematic review

OBJECTIVE

Artificial Intelligence (AI) offers potential opportunities to optimize clinical pharmacy services in community or hospital settings. The objective of this systematic literature review was to identify and analyse quantitative studies using or integrating AI for clinical pharmacy services.

MATERIALS AND METHODS

A systematic review was conducted using PubMed/Medline and Web of Science databases, including all articles published from 2000 to December 2021. Included studies had to involve pharmacists in the development or use of AI-powered apps and tools..

RESULTS

19 studies using AI for clinical pharmacy services were included in this review. 12 out of 19 articles (63.1%) were published in 2020 or 2021. Various methodologies of AI were used, mainly machine learning techniques and subsets (natural language processing and deep learning). The datasets used to train the models were mainly extracted from electronic medical records (6 studies, 32%). Among clinical pharmacy services, medication order review was the service most targeted by AI-powered apps and tools (9 studies), followed by health product dispensing (4 studies), pharmaceutical interviews and therapeutic education (2 studies). The development of these tools mainly involved hospital pharmacists (12/19 studies).

DISCUSSION AND CONCLUSION

The development of AI-powered apps and tools for clinical pharmacy services is just beginning. Pharmacists need to keep abreast of these developments in order to position themselves optimally while maintaining their human relationships with healthcare teams and patients. Significant efforts have to be made, in collaboration with data scientists, to better assess whether AI-powered apps and tools bring value to clinical pharmacy services in real practice.

Prediction of Prednisolone Dose Correction Using Machine Learning

Wrong dose, a common prescription error, can cause serious patient harm, especially in the case of high-risk drugs like oral corticosteroids. This study aims to build a machine learning model to predict dose-related prescription modifications for oral prednisolone tablets (i.e., highly imbalanced data with very few positive cases). Prescription data were obtained from the electronic medical records at a single institute. Cluster analysis classified the clinical departments into six clusters with similar patterns of prednisolone prescription. Two patterns of training datasets were created with/without preprocessing by the SMOTE method. Five ML models (SVM, KNN, GB, RF, and BRF) and logistic regression (LR) models were constructed by Python. The model was internally validated by five-fold stratified cross-validation and was validated with a 30% holdout test dataset. Eighty-two thousand five hundred fifty-three prescribing data for prednisolone tablets containing 135 dose-corrected positive cases were obtained. In the original dataset (without SMOTE), only the BRF model showed a good performance (in test dataset, ROC-AUC:0.917, recall: 0.951). In the training dataset preprocessed by SMOTE, performance was improved on all models. The highest performance models with SMOTE were SVM (in test dataset, ROC-AUC: 0.820, recall: 0.659) and BRF (ROC-AUC: 0.814, recall: 0.634). Although the prescribing data for dose-related collection are highly imbalanced, various techniques such as the following have allowed us to build high-performance prediction models: data preprocessing by SMOTE, stratified cross-validation, and BRF classifier corresponding to imbalanced data. ML is useful in complicated dose audits such as oral prednisolone.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41666-023-00128-3.

Development and validation of a machine learning-based detection system to improve precision screening for medication errors in the neonatal intensive care unit

Aim: To develop models that predict the presence of medication errors (MEs) (prescription, preparation, administration, and monitoring) using machine learning in NICU patients. Design: Prospective, observational cohort study randomized with machine learning (ML) algorithms. Setting: A 22-bed capacity NICU in Ankara, Turkey, between February 2020 and July 2021. Results: A total of 11,908 medication orders (28.9 orders/patient) for 412 NICU patients (5.53 drugs/patient/day) who received 2,280 prescriptions over 32,925 patient days were analyzed. At least one physician-related ME and nurse-related ME were found in 174 (42.2%) and 235 (57.0%) of the patients, respectively. The parameters that had the highest correlation with ME occurrence and subsequently included in the model were: total number of drugs, anti-infective drugs, nervous system drugs, 5-min APGAR score, postnatal age, alimentary tract and metabolism drugs, and respiratory system drugs as patient-related parameters, and weekly working hours of nurses, weekly working hours of physicians, and number of nurses' monthly shifts as care provider-related parameters. The obtained model showed high performance to predict ME (AUC: 0.920; 95% CI: 0.876-0.970) presence and is accessible online (http://softmed.hacettepe.edu.tr/NEO-DEER_Medication_Error/). Conclusion: This is the first developed and validated model to predict the presence of ME using work environment and pharmacotherapy parameters with high-performance ML algorithms in NICU patients. This approach and the current model hold the promise of implementation of targeted/precision screening to prevent MEs in neonates. Clinical Trial Registration: ClinicalTrials.gov, identifier NCT04899960.

Screening for Medication Errors and Adverse Events Using Outlier Detection Screening Algorithms in an Inpatient Setting

OBJECTIVES

To evaluate the potential of a novel system using outlier detection screening algorithms and to identify medication related risks in an inpatient setting.

METHODS

In the first phase of the study, we evaluated the transferability of models refined at another medical center using a different electronic medical record system (EMR) on 3 years of historical data (2017-2019), extracted from the local EMR system. Following the retrospective analysis, the system's models were fine-tuned to the specific local practice patterns. In the second, prospective phase of the study, the system was fully integrated in the local EMR and after a short run-in period was activated live. All alerts generated by the system, in both phases, were analyzed by a clinical team of physicians and pharmacists for accuracy and clinical relevance.

RESULTS

In the retrospective phase of the study, 226,804 medical orders were analyzed, generating a total of 2731 alerts (1.2% of medical orders). Of the alerts analyzed, 69% were clinically relevant alerts and 31% were false alerts. In the prospective phase of the study, 399 alerts were generated by the system (1.6% of medical orders). The vast majority of the alerts (72%) were considered clinically relevant, and 41% of the alerts caused a change in prescriber behavior (i.e. cancel/modify the medical order).

CONCLUSION

In an inpatient setting of a 600 bed computerized decision support system (CDSS) -naïve medical center, the system generated accurate and clinically valid alerts with low alert burden enabling physicians to improve daily medical practice.

Detecting Unusual Intravenous Infusion Alerting Patterns with Machine Learning Algorithms

OBJECTIVE

To detect unusual infusion alerting patterns using machine learning (ML) algorithms as a first step to advance safer inpatient intravenous administration of high-alert medications.

MATERIALS AND METHODS

We used one year of detailed propofol infusion data from a hospital. Interpretable and clinically relevant variables were feature engineered, and data points were aggregated per calendar day. A univariate (maximum times-limit) moving range (mr) control chart was used to simulate clinicians' common approach to identifying unusual infusion alerting patterns. Three different unsupervised multivariate ML-based anomaly detection algorithms (Local Outlier Factor, Isolation Forest, and k-Nearest Neighbors) were used for the same purpose. Results from the control chart and ML algorithms were compared.

RESULTS

The propofol data had 3,300 infusion alerts, 92% of which were generated during the day shift and seven of which had a times-limit greater than 10. The mr-chart identified 15 alert pattern anomalies. Different thresholds were set to include the top 15 anomalies from each ML algorithm. A total of 31 unique ML anomalies were grouped and ranked by agreeability. All algorithms agreed on 10% of the anomalies, and at least two algorithms agreed on 36%. Each algorithm detected one specific anomaly that the mr-chart did not detect. The anomaly represented a day with 71 propofol alerts (half of which were overridden) generated at an average rate of 1.06 per infusion, whereas the moving alert rate for the week was 0.35 per infusion.

DISCUSSION

These findings show that ML-based algorithms are more robust than control charts in detecting unusual alerting patterns. However, we recommend using a combination of algorithms, as multiple algorithms serve a benchmarking function and allow researchers to focus on data points with the highest algorithm agreeability.

CONCLUSION

Unsupervised ML algorithms can assist clinicians in identifying unusual alert patterns as a first step toward achieving safer infusion practices.

Assessment of a hybrid decision support system using machine learning with artificial intelligence to safely rule out prescriptions from medication review in daily practice

Background Medication review is time-consuming and not exhaustive in most French hospitals. We routinely use an innovative hybrid decision support system using Artificial Intelligence to prioritize medication review by scoring prescriptions by their risk of containing at least one drug related problem (DRP). Aim Our aim was to attest that the prescriptions with low risk of DRPs ruled out by the tool in everyday practice were effectively free of any DRPs with potentially severe clinical impact. Methods We conducted a randomized single-blinded study to compare the rate of pharmaceutical interventions (PI) between low and high-risk prescriptions defined by the tool's calculated score. Prescriptions were reviewed daily by a clinical pharmacist. Proportion of prescriptions with at least one severe DRP was calculated in both groups. Severe DRPs were characterized through a multidisciplinary approach. Results Four hundred and twenty (107 low score and 313 high score) prescriptions were analyzed. The percentage of prescriptions with severe DRPs was dramatically decreased in low score prescriptions (2.8% vs. 15.3% for high-risk; p = 0.0248). A significant difference was found (94% vs. 20%; p < 0.001) in the percentage of severe DRPs detected by the hybrid approach compared to a CDSS. During the study period, the hybrid tool allowed to rule out 55% of all prescriptions in our hospital.Conclusion This hybrid decision support tool has shown to be accurate to detect DRPs in daily practice. Despite some limitations, it offers the best possible solution to prioritized medication review, considering the shortage of clinical pharmacists in France and considerably improves the safety of patients' care.

Detecting Unusual Intravenous Infusion Alerting Patterns with Machine Learning Algorithms

OBJECTIVE

To detect unusual infusion alerting patterns using machine learning (ML) algorithms as a first step to advance safer inpatient intravenous administration of high-alert medications.

MATERIALS AND METHODS

We used one year of detailed propofol infusion data from a hospital. Interpretable and clinically relevant variables were feature engineered, and data points were aggregated per calendar day. A univariate (maximum times-limit) moving range (mr) control chart was used to simulate clinicians' common approach to identifying unusual infusion alerting patterns. Three different unsupervised multivariate ML-based anomaly detection algorithms (Local Outlier Factor, Isolation Forest, and k-Nearest Neighbors) were used for the same purpose. Results from the control chart and ML algorithms were compared.

RESULTS

The propofol data had 3,300 infusion alerts, 92% of which were generated during the day shift and seven of which had a times-limit greater than 10. The mr-chart identified 15 alert pattern anomalies. Different thresholds were set to include the top 15 anomalies from each ML algorithm. A total of 31 unique ML anomalies were grouped and ranked by agreeability. All algorithms agreed on 10% of the anomalies, and at least two algorithms agreed on 36%. Each algorithm detected one specific anomaly that the mr-chart did not detect. The anomaly represented a day with 71 propofol alerts (half of which were overridden) generated at an average rate of 1.06 per infusion, whereas the moving alert rate for the week was 0.35 per infusion.

DISCUSSION

These findings show that ML-based algorithms are more robust than control charts in detecting unusual alerting patterns. However, we recommend using a combination of algorithms, as multiple algorithms serve a benchmarking function and allow researchers to focus on data points with the highest algorithm agreeability.

CONCLUSION

Unsupervised ML algorithms can assist clinicians in identifying unusual alert patterns as a first step toward achieving safer infusion practices.

Clinical validation of clinical decision support systems for medication review: A scoping review

The aim of this scoping review is to summarize approaches and outcomes of clinical validation studies of clinical decision support systems (CDSSs) to support (part of) a medication review. A literature search was conducted in Embase and Medline. In total, 30 articles validating a CDSS were ultimately included. Most of the studies focused on detection of adverse drug events, potentially inappropriate medications and drug‐related problems. We categorized the included articles in three groups: studies subjectively reviewing the clinical relevance of CDSS's output (21/30 studies) resulting in a positive predictive value (PPV) for clinical relevance of 4–80%; studies determining the relationship between alerts and actual events (10/30 studies) resulting in a PPV for actual events of 5–80%; and studies comparing output of CDSSs to chart/medication reviews in the whole study population (10/30 studies) resulting in a sensitivity of 28–85% and specificity of 42–75%. We found heterogeneity in the methods used and in the outcome measures. The validation studies did not report the use of a published CDSS validation strategy. To improve the effectiveness and uptake of CDSSs supporting a medication review, future research would benefit from a more systematic and comprehensive validation strategy.

Proposal of a novel Artificial Intelligence Distribution Service platform for healthcare

In this paper, we focus on presenting a novel AI-based service platform proposal called AIDI (Artificial Intelligence Distribution Interface for healthcare). AIDI proposal is based on our earlier research work in which we evaluated AI-based healthcare services which have been used successfully in practice among healthcare service providers. We have also used our systematic review about AI-based healthcare services benefits in various healthcare sectors. This novel AIDI proposal contains services for health assessment, healthcare evaluation, and cognitive assistant which can be used by researchers, healthcare service provides, clinicians, and consumers. AIDI integrates multiple health databases and data lakes with AI service providers and open access AI algorithms. It also gives healthcare service providers open access to state-of-the-art AI-based diagnosis and analysis services. This paper provides a description of AIDI platform, how it could be developed, what can become obstacles in the development, and how the platform can provide benefits to healthcare when it will be operational in the future.

Key use cases for artificial intelligence to reduce the frequency of adverse drug events: a scoping review

Adverse drug events (ADEs) represent one of the most prevalent types of health-care-related harm, and there is substantial room for improvement in the way that they are currently predicted and detected. We conducted a scoping review to identify key use cases in which artificial intelligence (AI) could be leveraged to reduce the frequency of ADEs. We focused on modern machine learning techniques and natural language processing. 78 articles were included in the scoping review. Studies were heterogeneous and applied various AI techniques covering a wide range of medications and ADEs. We identified several key use cases in which AI could contribute to reducing the frequency and consequences of ADEs, through prediction to prevent ADEs and early detection to mitigate the effects. Most studies (73 [94%] of 78) assessed technical algorithm performance, and few studies evaluated the use of AI in clinical settings. Most articles (58 [74%] of 78) were published within the past 5 years, highlighting an emerging area of study. Availability of new types of data, such as genetic information, and access to unstructured clinical notes might further advance the field.

Detection of overdose and underdose prescriptions-An unsupervised machine learning approach

Overdose prescription errors sometimes cause serious life-threatening adverse drug events, while underdose errors lead to diminished therapeutic effects. Therefore, it is important to detect and prevent these errors. In the present study, we used the one-class support vector machine (OCSVM), one of the most common unsupervised machine learning algorithms for anomaly detection, to identify overdose and underdose prescriptions. We extracted prescription data from electronic health records in Kyushu University Hospital between January 1, 2014 and December 31, 2019. We constructed an OCSVM model for each of the 21 candidate drugs using three features: age, weight, and dose. Clinical overdose and underdose prescriptions, which were identified and rectified by pharmacists before administration, were collected. Synthetic overdose and underdose prescriptions were created using the maximum and minimum doses, defined by drug labels or the UpToDate database. We applied these prescription data to the OCSVM model and evaluated its detection performance. We also performed comparative analysis with other unsupervised outlier detection algorithms (local outlier factor, isolation forest, and robust covariance). Twenty-seven out of 31 clinical overdose and underdose prescriptions (87.1%) were detected as abnormal by the model. The constructed OCSVM models showed high performance for detecting synthetic overdose prescriptions (precision 0.986, recall 0.964, and F-measure 0.973) and synthetic underdose prescriptions (precision 0.980, recall 0.794, and F-measure 0.839). In comparative analysis, OCSVM showed the best performance. Our models detected the majority of clinical overdose and underdose prescriptions and demonstrated high performance in synthetic data analysis. OCSVM models, constructed using features such as age, weight, and dose, are useful for detecting overdose and underdose prescriptions.

Recent Developments in Privacy-Preserving Mining of Clinical Data

With the dramatic increases in both the capability to collect personal data and the capability to analyze large amounts of data, increasingly sophisticated and personal insights are being drawn. These insights are valuable for clinical applications but also open up possibilities for identification and abuse of personal information. In this paper, we survey recent research on classical methods of privacy-preserving data mining. Looking at dominant techniques and recent innovations to them, we examine the applicability of these methods to the privacy-preserving analysis of clinical data. We also discuss promising directions for future research in this area.

High alert drugs screening using gradient boosting classifier

Prescription errors in high alert drugs (HAD), a group of drugs that have a high risk of complications and potential negative consequences, are a major and serious problem in medicine. Standardized hospital interventions, protocols, or guidelines were implemented to reduce the errors but were not found to be highly effective. Machine learning driven clinical decision support systems (CDSS) show a potential solution to address this problem. We developed a HAD screening protocol with a machine learning model using Gradient Boosting Classifier and screening parameters to identify the events of HAD prescription errors from the drug prescriptions of out and inpatients at Maharaj Nakhon Chiang Mai hospital in 2018. The machine learning algorithm was able to screen drug prescription events with a risk of HAD inappropriate use and identify over 98% of actual HAD mismatches in the test set and 99% in the evaluation set. This study demonstrates that machine learning plays an important role and has potential benefit to screen and reduce errors in HAD prescriptions.

AI in healthcare: A narrative review

In this paper, we focus on providing a narrative review of healthcare services in which artificial intelligence (AI) based services are used as part of the operations and analyze key elements to create successful AI-based services for healthcare. The benefits of AI in healthcare are measured by how AI is improving the healthcare outcomes, help caregivers in work, and reducing healthcare costs. AI market in healthcare sector have also a high market potential with 28% global compound annual growth rate. This paper will collect outcomes from multiple perspectives of healthcare sector including financial, health improvement, and care outcome as well as provide proposals and key factors for successful implementation of AI methods in healthcare. It is shown in this paper that AI implementation in healthcare can provide cost reduction and same time provide better health outcome for all.

Associations of physicians' prescribing experience, work hours, and workload with prescription errors

OBJECTIVE

We aimed to assess associations of physician's work overload, successive work shifts, and work experience with physicians' risk to err.

MATERIALS AND METHODS

This large-scale study included physicians who prescribed at least 100 systemic medications at Sheba Medical Center during 2012-2017 in all acute care departments, excluding intensive care units. Presumed medication errors were flagged by a high-accuracy computerized decision support system that uses machine-learning algorithms to detect potential medication prescription errors. Physicians' successive work shifts (first or only shift, second, and third shifts), workload (assessed by the number of prescriptions during a shift) and work-experience, as well as a novel measurement of physicians' prescribing experience with a specific drug, were assessed per prescription. The risk to err was determined for various work conditions.

RESULTS

1 652 896 medical orders were prescribed by 1066 physicians; The system flagged 3738 (0.23%) prescriptions as erroneous. Physicians were 8.2 times more likely to err during high than normal-low workload shifts (5.19% vs 0.63%, P < .0001). Physicians on their third or second successive shift (compared to a first or single shift) were more likely to err (2.1%, 1.8%, and 0.88%, respectively, P < .001). Lack of experience in prescribing a specific medication was associated with higher error rate (0.37% for the first 5 prescriptions vs 0.13% after over 40, P < .001).

DISCUSSION

Longer hours and less experience in prescribing a specific medication increase risk of erroneous prescribing.

CONCLUSION

Restricting successive shifts, reducing workload, increasing training and supervision, and implementing smart clinical decision support systems may help reduce prescription errors.

Pharmacists' perceptions of a machine learning model for the identification of atypical medication orders

OBJECTIVES

The study sought to assess the clinical performance of a machine learning model aiming to identify unusual medication orders.

MATERIALS AND METHODS

This prospective study was conducted at CHU Sainte-Justine, Canada, from April to August 2020. An unsupervised machine learning model based on GANomaly and 2 baselines were trained to learn medication order patterns from 10 years of data. Clinical pharmacists dichotomously (typical or atypical) labeled orders and pharmacological profiles (patients' medication lists). Confusion matrices, areas under the precision-recall curve (AUPRs), and F1 scores were calculated.

RESULTS

A total of 12 471 medication orders and 1356 profiles were labeled by 25 pharmacists. Medication order predictions showed a precision of 35%, recall (sensitivity) of 26%, and specificity of 97% as compared with pharmacist labels, with an AUPR of 0.25 and an F1 score of 0.30. Profile predictions showed a precision of 49%, recall of 75%, and specificity of 82%, with an AUPR of 0.60, and an F1 score of 0.59. The model performed better than the baselines. According to the pharmacists, the model was a useful screening tool, and 9 of 15 participants preferred predictions by medication, rather than by profile.

DISCUSSION

Predictions for profiles had higher F1 scores and recall compared with medication order predictions. Although the performance was much better for profile predictions, pharmacists generally preferred medication order predictions.

CONCLUSIONS

Based on the AUPR, this model showed better performance for the identification of atypical pharmacological profiles than for medication orders. Pharmacists considered the model a useful screening tool. Improving these predictions should be prioritized in future research to maximize clinical impact.

MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care

The rise of digital data and computing power have contributed to significant advancements in artificial intelligence (AI), leading to the use of classification and prediction models in health care to enhance clinical decision-making for diagnosis, treatment and prognosis. However, such advances are limited by the lack of reporting standards for the data used to develop those models, the model architecture, and the model evaluation and validation processes. Here, we present MINIMAR (MINimum Information for Medical AI Reporting), a proposal describing the minimum information necessary to understand intended predictions, target populations, and hidden biases, and the ability to generalize these emerging technologies. We call for a standard to accurately and responsibly report on AI in health care. This will facilitate the design and implementation of these models and promote the development and use of associated clinical decision support tools, as well as manage concerns regarding accuracy and bias.

[Not Available]

BACKGROUND

Artificial intelligence (AI) can be described as an advanced technology in which machines display a certain form of intelligence.

OBJECTIVES

The primary objective was to perform a narrative review of studies evaluating the feasibility and impact of AI in pharmacy. The secondary objective was to create a mind map of AI in health care.

DATA SOURCES

Four databases were consulted: PubMed, Medline, Embase, and CINAHL.

STUDY SELECTION AND DATA EXTRACTION

Four search strategies were developed. Initial selection of articles was based on their titles and abstracts; the full texts were then evaluated by a research assistant, with review by a pharmacist. Articles were included if they described or evaluated the feasibility or impact of AI in pharmacy.

DATA SYNTHESIS

A total of 362 articles were identified by the literature review, of which 18 met the inclusion criteria. The studies were mainly conducted in the United States (72%, 13/18). The article topics were, in decreasing order, prediction of response to treatments and adverse effects (33%, 6/18), patient prioritization (28%, 5/18), treatment adherence (22%, 4/18), validation of prescriptions and electronic prescription (17%, 3/18), and other themes (e.g., diagnosis, costs, insurance, and verification of syringe volume).

CONCLUSIONS

This narrative review highlighted 18 studies evaluating the feasibility and impact of AI in pharmacy. The studies used various methodologies in different settings, both retail pharmacies and hospital pharmacies. It is still too soon to predict the implications of AI for pharmacy, but these studies emphasize the importance of attention in this area.

Reducing drug prescription errors and adverse drug events by application of a probabilistic, machine-learning based clinical decision support system in an inpatient setting

BACKGROUND

Drug prescription errors are made, worldwide, on a daily basis, resulting in a high burden of morbidity and mortality. Existing rule-based systems for prevention of such errors are unsuccessful and associated with substantial burden of false alerts.

OBJECTIVE

In this prospective study, we evaluated the accuracy, validity, and clinical usefulness of medication error alerts generated by a novel system using outlier detection screening algorithms, used on top of a legacy standard system, in a real-life inpatient setting.

MATERIALS AND METHODS

We integrated a novel outlier system into an existing electronic medical record system, in a single medical ward in a tertiary medical center. The system monitored all drug prescriptions written during 16 months. The department's staff assessed all alerts for accuracy, clinical validity, and usefulness. We recorded all physician's real-time responses to alerts generated.

RESULTS

The alert burden generated by the system was low, with alerts generated for 0.4% of all medication orders. Sixty percent of the alerts were flagged after the medication was already dispensed following changes in patients' status which necessitated medication changes (eg, changes in vital signs). Eighty-five percent of the alerts were confirmed clinically valid, and 80% were considered clinically useful. Forty-three percent of the alerts caused changes in subsequent medical orders.

CONCLUSION

A clinical decision support system that used a probabilistic, machine-learning approach based on statistically derived outliers to detect medication errors generated clinically useful alerts. The system had high accuracy, low alert burden and low false-positive rate, and led to changes in subsequent orders.

Assessing the International Transferability of a Machine Learning Model for Detecting Medication Error in the General Internal Medicine Clinic: Multicenter Preliminary Validation Study

Background

Although most current medication error prevention systems are rule-based, these systems may result in alert fatigue because of poor accuracy. Previously, we had developed a machine learning (ML) model based on Taiwan’s local databases (TLD) to address this issue. However, the international transferability of this model is unclear.

Objective

This study examines the international transferability of a machine learning model for detecting medication errors and whether the federated learning approach could further improve the accuracy of the model.

Methods

The study cohort included 667,572 outpatient prescriptions from 2 large US academic medical centers. Our ML model was applied to build the original model (O model), the local model (L model), and the hybrid model (H model). The O model was built using the data of 1.34 billion outpatient prescriptions from TLD. A validation set with 8.98% (60,000/667,572) of the prescriptions was first randomly sampled, and the remaining 91.02% (607,572/667,572) of the prescriptions served as the local training set for the L model. With a federated learning approach, the H model used the association values with a higher frequency of co-occurrence among the O and L models. A testing set with 600 prescriptions was classified as substantiated and unsubstantiated by 2 independent physician reviewers and was then used to assess model performance.

Results

The interrater agreement was significant in terms of classifying prescriptions as substantiated and unsubstantiated (κ=0.91; 95% CI 0.88 to 0.95). With thresholds ranging from 0.5 to 1.5, the alert accuracy ranged from 75%-78% for the O model, 76%-78% for the L model, and 79%-85% for the H model.

Conclusions

Our ML model has good international transferability among US hospital data. Using the federated learning approach with local hospital data could further improve the accuracy of the model.

Artificial intelligence enabled applications in kidney disease

Artificial intelligence (AI) is considered as the next natural progression of traditional statistical techniques. Advances in analytical methods and infrastructure enable AI to be applied in health care. While AI applications are relatively common in fields like ophthalmology and cardiology, its use is scarcely reported in nephrology. We present the current status of AI in research toward kidney disease and discuss future pathways for AI. The clinical applications of AI in progression to end-stage kidney disease and dialysis can be broadly subdivided into three main topics: (a) predicting events in the future such as mortality and hospitalization; (b) providing treatment and decision aids such as automating drug prescription; and (c) identifying patterns such as phenotypical clusters and arteriovenous fistula aneurysm. At present, the use of prediction models in treating patients with kidney disease is still in its infancy and further evidence is needed to identify its relative value. Policies and regulations need to be addressed before implementing AI solutions at the point of care in clinics. AI is not anticipated to replace the nephrologists' medical decision-making, but instead assist them in providing optimal personalized care for their patients.

Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review

BACKGROUND

Artificial intelligence (AI) provides opportunities to identify the health risks of patients and thus influence patient safety outcomes.

OBJECTIVE

The purpose of this systematic literature review was to identify and analyze quantitative studies utilizing or integrating AI to address and report clinical-level patient safety outcomes.

METHODS

We restricted our search to the PubMed, PubMed Central, and Web of Science databases to retrieve research articles published in English between January 2009 and August 2019. We focused on quantitative studies that reported positive, negative, or intermediate changes in patient safety outcomes using AI apps, specifically those based on machine-learning algorithms and natural language processing. Quantitative studies reporting only AI performance but not its influence on patient safety outcomes were excluded from further review.

RESULTS

We identified 53 eligible studies, which were summarized concerning their patient safety subcategories, the most frequently used AI, and reported performance metrics. Recognized safety subcategories were clinical alarms (n=9; mainly based on decision tree models), clinical reports (n=21; based on support vector machine models), and drug safety (n=23; mainly based on decision tree models). Analysis of these 53 studies also identified two essential findings: (1) the lack of a standardized benchmark and (2) heterogeneity in AI reporting.

CONCLUSIONS

This systematic review indicates that AI-enabled decision support systems, when implemented correctly, can aid in enhancing patient safety by improving error detection, patient stratification, and drug management. Future work is still needed for robust validation of these systems in prospective and real-world clinical environments to understand how well AI can predict safety outcomes in health care settings.

Artificial intelligence in sleep medicine: background and implications for clinicians

None

Polysomnography remains the cornerstone of objective testing in sleep medicine and results in massive amounts of electrophysiological data, which is well-suited for analysis with artificial intelligence (AI)-based tools. Combined with other sources of health data, AI is expected to provide new insights to inform the clinical care of sleep disorders and advance our understanding of the integral role sleep plays in human health. Additionally, AI has the potential to streamline day-to-day operations and therefore optimize direct patient care by the sleep disorders team. However, clinicians, scientists, and other stakeholders must develop best practices to integrate this rapidly evolving technology into our daily work while maintaining the highest degree of quality and transparency in health care and research. Ultimately, when harnessed appropriately in conjunction with human expertise, AI will improve the practice of sleep medicine and further sleep science for the health and well-being of our patients.

Predicting Inpatient Medication Orders From Electronic Health Record Data

In a general inpatient population, we predicted patient‐specific medication orders based on structured information in the electronic health record (EHR). Data on over three million medication orders from an academic medical center were used to train two machine‐learning models: A deep learning sequence model and a logistic regression model. Both were compared with a baseline that ranked the most frequently ordered medications based on a patient’s discharge hospital service and amount of time since admission. Models were trained to predict from 990 possible medications at the time of order entry. Fifty‐five percent of medications ordered by physicians were ranked in the sequence model’s top‐10 predictions (logistic model: 49%) and 75% ranked in the top‐25 (logistic model: 69%). Ninety‐three percent of the sequence model’s top‐10 prediction sets contained at least one medication that physicians ordered within the next day. These findings demonstrate that medication orders can be predicted from information present in the EHR.

[Information technology and eHealth to improve patient safety]

Patient safety is a key element of high-quality healthcare. Digitalization, new eHealth applications and data-based algorithms have high potential to make a significant contribution. This article presents current technological developments along a simplified patient journey from emergency medical triage, diagnosis and therapy to follow-up. The technical interventions are highly diverse and mostly accompanied by a low level of evidence, since most of them are from single academic projects or start-ups. Although there should be no doubt that technology is an important instrument for increasing patient safety, new technologies also involve new risks. Furthermore, technical measures must always be embedded in an overall concept of organizational measures, adequate education, training and accompanying research in order to generate the highest possible benefits and lowest possible risks.

Using a Machine Learning System to Identify and Prevent Medication Prescribing Errors: A Clinical and Cost Analysis Evaluation

BACKGROUND

Clinical decision support (CDS) alerting tools can identify and reduce medication errors. However, they are typically rule-based and can identify only the errors previously programmed into their alerting logic. Machine learning holds promise for improving medication error detection and reducing costs associated with adverse events. This study evaluates the ability of a machine learning system (MedAware) to generate clinically valid alerts and estimates the cost savings associated with potentially prevented adverse events.

METHODS

Alerts were generated retrospectively by the MedAware system on outpatient data from two academic medical centers between 2009 and 2013. MedAware alerts were compared to alerts in an existing CDS system. A random sample of 300 alerts was selected for medical record review. Frequency and severity of potential outcomes of alerted medication errors of medium and high clinical value were estimated, along with associated health care costs of these potentially prevented adverse events.

RESULTS

A total of 10,668 alerts were generated. Overall, 68.2% of MedAware alerts would not have been generated by the existing CDS system. Ninety-two percent of a random sample of the chart-reviewed alerts were accurate based on structured data available in the record, and 79.7% were clinically valid. Estimated cost of adverse events potentially prevented in an outpatient setting was more than $60 per drug alert and $1.3 million when extrapolating study findings to the full patient population.

CONCLUSION

A machine learning system identified clinically valid medication error alerts that might otherwise be missed with existing CDS systems. Estimates show potential for cost savings associated with potentially prevented adverse events.

How to Accurately and Privately Identify Anomalies

Identifying anomalies in data is central to the advancement of science, national security, and finance. However, privacy concerns restrict our ability to analyze data. Can we lift these restrictions and accurately identify anomalies without hurting the privacy of those who contribute their data? We address this question for the most practically relevant case, where a record is considered anomalous relative to other records. We make four contributions. First, we introduce the notion of sensitive privacy, which conceptualizes what it means to privately identify anomalies. Sensitive privacy generalizes the important concept of differential privacy and is amenable to analysis. Importantly, sensitive privacy admits algorithmic constructions that provide strong and practically meaningful privacy and utility guarantees. Second, we show that differential privacy is inherently incapable of accurately and privately identifying anomalies; in this sense, our generalization is necessary. Third, we provide a general compiler that takes as input a differentially private mechanism (which has bad utility for anomaly identification) and transforms it into a sensitively private one. This compiler, which is mostly of theoretical importance, is shown to output a mechanism whose utility greatly improves over the utility of the input mechanism. As our fourth contribution we propose mechanisms for a popular definition of anomaly ((β, r)-anomaly) that (i) are guaranteed to be sensitively private, (ii) come with provable utility guarantees, and (iii) are empirically shown to have an overwhelmingly accurate performance over a range of datasets and evaluation criteria.

Automated detection of wrong-drug prescribing errors

BACKGROUND

To assess the specificity of an algorithm designed to detect look-alike/sound-alike (LASA) medication prescribing errors in electronic health record (EHR) data.

SETTING

Urban, academic medical centre, comprising a 495-bed hospital and outpatient clinic running on the Cerner EHR. We extracted 8 years of medication orders and diagnostic claims. We licensed a database of medication indications, refined it and merged it with the medication data. We developed an algorithm that triggered for LASA errors based on name similarity, the frequency with which a patient received a medication and whether the medication was justified by a diagnostic claim. We stratified triggers by similarity. Two clinicians reviewed a sample of charts for the presence of a true error, with disagreements resolved by a third reviewer. We computed specificity, positive predictive value (PPV) and yield.

RESULTS

The algorithm analysed 488 481 orders and generated 2404 triggers (0.5% rate). Clinicians reviewed 506 cases and confirmed the presence of 61 errors, for an overall PPV of 12.1% (95% CI 10.7% to 13.5%). It was not possible to measure sensitivity or the false-negative rate. The specificity of the algorithm varied as a function of name similarity and whether the intended and dispensed drugs shared the same route of administration.

CONCLUSION

Automated detection of LASA medication errors is feasible and can reveal errors not currently detected by other means. Real-time error detection is not possible with the current system, the main barrier being the real-time availability of accurate diagnostic information. Further development should replicate this analysis in other health systems and on a larger set of medications and should decrease clinician time spent reviewing false-positive triggers by increasing specificity.

Artificial Intelligence in Clinical Decision Support: a Focused Literature Survey

OBJECTIVES

This survey analyses the latest literature contributions to clinical decision support systems (DSSs) on a two-year period (2017-2018), focusing on the approaches that adopt Artificial Intelligence (AI) techniques in a broad sense. The goal is to analyse the distribution of data-driven AI approaches with respect to "classical" knowledge-based ones, and to consider the issues raised and their possible solutions.

METHODS

We included PubMed and Web of ScienceTM publications, focusing on contributions describing clinical DSSs that adopted one or more AI methodologies.

RESULTS

We selected 75 papers, 49 of which describe approaches in the data-driven AI area, 20 present purely knowledge-based DSSs, and 6 adopt hybrid approaches relying on both formalized knowledge and data.

CONCLUSIONS

Recent studies in the clinical DSS area demonstrate a prevalence of data-driven AI, which can be adopted autonomously in purely data-driven systems, or in cooperation with domain knowledge in hybrid systems. Such hybrid approaches, able to conjugate all available knowledge sources through proper knowledge integration steps, represent an interesting example of synergy between the two AI categories. This synergy can lead to the resolution of some existing issues, such as the need for transparency and explainability, nowadays recognized as central themes to be addressed by both AI and medical informatics research.

Exploring the Impact of the Prescription Automatic Screening System in Health Care Services: Quasi-Experiment

BACKGROUND

Hospitals have deployed various types of technologies to alleviate the problem of high medical costs. The cost of pharmaceuticals is one of the main drivers of medical costs. The Prescription Automatic Screening System (PASS) aims to monitor physicians' prescribing behavior, which has the potential to decrease prescription errors and medical treatment costs. However, a substantial number of cases with unsatisfactory results related to the effects of PASS have been noted.

OBJECTIVE

The objectives of this study were to systematically explore the imperative role of PASS on hospitals' prescription errors and medical treatment costs and examine its contingency factors to clarify the various factors associated with the effective use of PASS.

METHODS

To systematically examine the various effects of PASS, we adopted a quasi-experiment methodology by using a 2-year observation dataset from 2 hospitals in China. We then analyzed the data related to physicians' prescriptions both before and after the deployment of PASS and eliminated influences from a variety of perplexing factors by utilizing a control hospital that did not use a PASS system. In total, 754 physicians were included in this experiment comprising 11,054 patients: 400 physicians in the treatment group and 354 physicians in the control group. This study was also preceded by a series of interviews, which were employed to identify moderators. Thereafter, we adopted propensity score matching integrated with difference-in-differences to isolate the effects of PASS.

RESULTS

The effects of PASS on prescription errors and medical treatment costs were all significant (error: 95% CI -0.40 to -0.11, P=.001; costs: 95% CI -0.75 to -0.12, P=.007). Pressure from organizational rules and workload decreased the effect of PASS on prescription errors (95% CI 0.18-0.39; P<.001) and medical treatment costs (95% CI 0.07-0.55; P=.01), respectively. We also suspected that other pressures (eg, clinical title and risk categories of illness) also impaired physicians' attention to alerts from PASS. However, the effects of PASS did not change among physicians with a higher clinical title or when treating diseases demonstrating high risk. This may be attributed to the fact that these physicians will focus more on their patients in these situations, regardless of having access to an intelligent system.

CONCLUSIONS

Although implementation of PASS decreases prescription errors and medical treatment costs, workload and organizational rules remain problematic, as they tend to impair the positive effects of auxiliary diagnosis systems on performance. This again highlights the importance of considering both technical and organizational issues to obtain the highest level of effectiveness when deploying information technology in hospitals.

Better medicine through machine learning: What's real, and what's artificial?

Machine Learning Special Issue Guest Editors Suchi Saria, Atul Butte, and Aziz Sheikh cut through the hyperbole with an accessible and accurate portrayal of the forefront of machine learning in clinical translation.

Strategies for improving physician documentation in the emergency department: a systematic review

BACKGROUND

Physician chart documentation can facilitate patient care decisions, reduce treatment errors, and inform health system planning and resource allocation activities. Although accurate and complete patient chart data supports quality and continuity of patient care, physician documentation often varies in terms of timeliness, legibility, clarity and completeness. While many educational and other approaches have been implemented in hospital settings, the extent to which these interventions can improve the quality of documentation in emergency departments (EDs) is unknown.

METHODS

We conducted a systematic review to assess the effectiveness of approaches to improve ED physician documentation. Peer reviewed electronic databases, grey literature sources, and reference lists of included studies were searched to March 2015. Studies were included if they reported on outcomes associated with interventions designed to enhance the quality of physician documentation.

RESULTS

Nineteen studies were identified that report on the effectiveness of interventions to improve physician documentation in EDs. Interventions included audit/feedback, dictation, education, facilitation, reminders, templates, and multi-interventions. While ten studies found that audit/feedback, dictation, pharmacist facilitation, reminders, templates, and multi-pronged approaches did improve the quality of physician documentation across multiple outcome measures, the remaining nine studies reported mixed results.

CONCLUSIONS

Promising approaches to improving physician documentation in emergency department settings include audit/feedback, reminders, templates, and multi-pronged education interventions. Future research should focus on exploring the impact of implementing these interventions in EDs with and without emergency medical record systems (EMRs), and investigating the potential of emerging technologies, including EMR-based machine-learning, to promote improvements in the quality of ED documentation.

DDC-Outlier: Preventing Medication Errors Using Unsupervised Learning

Electronic health records have brought valuable improvements to hospital practices by integrating patient information. In fact, the understanding of these data can prevent mistakes that may put patients' lives at risk. Nonetheless, to the best of our knowledge, there are no previous studies addressing the automatic detection of outlier prescriptions, regarding dosage and frequency. In this paper, we propose an unsupervised method, called density-distance-centrality (DDC), to detect potential outlier prescriptions. A dataset with 563 thousand prescribed medications was used to assess our proposed approach against different state-of-the-art techniques for outlier detection. In the experiments, our approach achieves better results in the task of overdose and underdose detection in medical prescriptions, compared to other methods applied to this problem. Additionally, most of the false positive instances detected by our algorithm were potential prescriptions errors.