A Machine Learning Approach with Human-AI Collaboration for Automated Classification of Patient Safety Event Reports: Algorithm Development and Validation Study

BACKGROUND

Adverse events refer to incidents with potential or actual harm to patients in hospitals. These events are typically documented through patient safety event (PSE) reports, which consist of detailed narratives providing contextual information on the occurrences. Accurate classification of PSE reports is crucial for patient safety monitoring. However, this process faces challenges due to inconsistencies in classifications and the sheer volume of reports. Recent advancements in text representation, particularly contextual text representation derived from transformer-based language models, offer a promising solution for more precise PSE report classification. Integrating the machine learning (ML) classifier necessitates a balance between human expertise and artificial intelligence (AI). Central to this integration is the concept of explainability, which is crucial for building trust and ensuring effective human-AI collaboration.

OBJECTIVE

This study aims to investigate the efficacy of ML classifiers trained using contextual text representation in automatically classifying PSE reports. Furthermore, the study presents an interface that integrates the ML classifier with the explainability technique to facilitate human-AI collaboration for PSE report classification.

METHODS

This study used a data set of 861 PSE reports from a large academic hospital's maternity units in the Southeastern United States. Various ML classifiers were trained with both static and contextual text representations of PSE reports. The trained ML classifiers were evaluated with multiclass classification metrics and the confusion matrix. The local interpretable model-agnostic explanations (LIME) technique was used to provide the rationale for the ML classifier's predictions. An interface that integrates the ML classifier with the LIME technique was designed for incident reporting systems.

RESULTS

The top-performing classifier using contextual representation was able to obtain an accuracy of 75.4% (95/126) compared to an accuracy of 66.7% (84/126) by the top-performing classifier trained using static text representation. A PSE reporting interface has been designed to facilitate human-AI collaboration in PSE report classification. In this design, the ML classifier recommends the top 2 most probable event types, along with the explanations for the prediction, enabling PSE reporters and patient safety analysts to choose the most suitable one. The LIME technique showed that the classifier occasionally relies on arbitrary words for classification, emphasizing the necessity of human oversight.

CONCLUSIONS

This study demonstrates that training ML classifiers with contextual text representations can significantly enhance the accuracy of PSE report classification. The interface designed in this study lays the foundation for human-AI collaboration in the classification of PSE reports. The insights gained from this research enhance the decision-making process in PSE report classification, enabling hospitals to more efficiently identify potential risks and hazards and enabling patient safety analysts to take timely actions to prevent patient harm.

Interpretable Detection of Diabetic Retinopathy, Retinal Vein Occlusion, Age-Related Macular Degeneration, and Other Fundus Conditions

Diabetic retinopathy (DR), retinal vein occlusion (RVO), and age-related macular degeneration (AMD) pose significant global health challenges, often resulting in vision impairment and blindness. Automatic detection of these conditions is crucial, particularly in underserved rural areas with limited access to ophthalmic services. Despite remarkable advancements in artificial intelligence, especially convolutional neural networks (CNNs), their complexity can make interpretation difficult. In this study, we curated a dataset consisting of 15,089 color fundus photographs (CFPs) obtained from 8110 patients who underwent fundus fluorescein angiography (FFA) examination. The primary objective was to construct integrated models that merge CNNs with an attention mechanism. These models were designed for a hierarchical multilabel classification task, focusing on the detection of DR, RVO, AMD, and other fundus conditions. Furthermore, our approach extended to the detailed classification of DR, RVO, and AMD according to their respective subclasses. We employed a methodology that entails the translation of diagnostic information obtained from FFA results into CFPs. Our investigation focused on evaluating the models' ability to achieve precise diagnoses solely based on CFPs. Remarkably, our models showcased improvements across diverse fundus conditions, with the ConvNeXt-base + attention model standing out for its exceptional performance. The ConvNeXt-base + attention model achieved remarkable metrics, including an area under the receiver operating characteristic curve (AUC) of 0.943, a referable F1 score of 0.870, and a Cohen's kappa of 0.778 for DR detection. For RVO, it attained an AUC of 0.960, a referable F1 score of 0.854, and a Cohen's kappa of 0.819. Furthermore, in AMD detection, the model achieved an AUC of 0.959, an F1 score of 0.727, and a Cohen's kappa of 0.686. Impressively, the model demonstrated proficiency in subclassifying RVO and AMD, showcasing commendable sensitivity and specificity. Moreover, our models enhanced interpretability by visualizing attention weights on fundus images, aiding in the identification of disease findings. These outcomes underscore the substantial impact of our models in advancing the detection of DR, RVO, and AMD, offering the potential for improved patient outcomes and positively influencing the healthcare landscape.

Explainable deep learning enhances robust and reliable real-time monitoring of a chromatographic protein A capture step

The application of model-based real-time monitoring in biopharmaceutical production is a major step toward quality-by-design and the fundament for model predictive control. Data-driven models have proven to be a viable option to model bioprocesses. In the high stakes setting of biopharmaceutical manufacturing it is essential to ensure high model accuracy, robustness, and reliability. That is only possible when (i) the data used for modeling is of high quality and sufficient size, (ii) state-of-the-art modeling algorithms are employed, and (iii) the input-output mapping of the model has been characterized. In this study, we evaluate the accuracy of multiple data-driven models in predicting the monoclonal antibody (mAb) concentration, double stranded DNA concentration, host cell protein concentration, and high molecular weight impurity content during elution from a protein A chromatography capture step. The models achieved high-quality predictions with a normalized root mean squared error of <4% for the mAb concentration and of ≈10% for the other process variables. Furthermore, we demonstrate how permutation/occlusion-based methods can be used to gain an understanding of dependencies learned by one of the most complex data-driven models, convolutional neural network ensembles. We observed that the models generally exhibited dependencies on correlations that agreed with first principles knowledge, thereby bolstering confidence in model reliability. Finally, we present a workflow to assess the model behavior in case of systematic measurement errors that may result from sensor fouling or failure. This study represents a major step toward improved viability of data-driven models in biopharmaceutical manufacturing.

An Interpretable Three-Dimensional Artificial Intelligence Model for Computer-Aided Diagnosis of Lung Nodules in Computed Tomography Images

Lung cancer is typically classified into small-cell carcinoma and non-small-cell carcinoma. Non-small-cell carcinoma accounts for approximately 85% of all lung cancers. Low-dose chest computed tomography (CT) can quickly and non-invasively diagnose lung cancer. In the era of deep learning, an artificial intelligence (AI) computer-aided diagnosis system can be developed for the automatic recognition of CT images of patients, creating a new form of intelligent medical service. For many years, lung cancer has been the leading cause of cancer-related deaths in Taiwan, with smoking and air pollution increasing the likelihood of developing the disease. The incidence of lung adenocarcinoma in never-smoking women has also increased significantly in recent years, resulting in an important public health problem. Early detection of lung cancer and prompt treatment can help reduce the mortality rate of patients with lung cancer. In this study, an improved 3D interpretable hierarchical semantic convolutional neural network named HSNet was developed and validated for the automatic diagnosis of lung cancer based on a collection of lung nodule images. The interpretable AI model proposed in this study, with different training strategies and adjustment of model parameters, such as cyclic learning rate and random weight averaging, demonstrated better diagnostic performance than the previous literature, with results of a four-fold cross-validation procedure showing calcification: 0.9873 ± 0.006, margin: 0.9207 ± 0.009, subtlety: 0.9026 ± 0.014, texture: 0.9685 ± 0.006, sphericity: 0.8652 ± 0.021, and malignancy: 0.9685 ± 0.006.

Intelligent Fault Diagnosis of Liquid Rocket Engine via Interpretable LSTM with Multisensory Data

Fault diagnosis is essential for high energy systems such as liquid rocket engines (LREs) due to harsh thermal and mechanical working environment. In this study, a novel method based on one-dimension Convolutional Neural Network (1D-CNN) and interpretable bidirectional Long Short-term Memory (LSTM) is proposed for intelligent fault diagnosis of LREs. 1D-CNN is responsible for extracting sequential signals collected from multi sensors. Then the interpretable LSTM is developed to model the extracted features, which contributes to modeling the temporal information. The proposed method was executed for fault diagnosis using the simulated measurement data of the LRE mathematical model. The results demonstrate the proposed algorithm outperforms other methods in terms of accuracy of fault diagnosis. Through experimental verification, the method proposed in this paper was compared with CNN, 1DCNN-SVM and CNN-LSTM in terms of LRE startup transient fault recognition performance. The model proposed in this paper had the highest fault recognition accuracy (97.39%).

Interpretable Machine Learning Techniques in ECG-Based Heart Disease Classification: A Systematic Review

Heart disease is one of the leading causes of mortality throughout the world. Among the different heart diagnosis techniques, an electrocardiogram (ECG) is the least expensive non-invasive procedure. However, the following are challenges: the scarcity of medical experts, the complexity of ECG interpretations, the manifestation similarities of heart disease in ECG signals, and heart disease comorbidity. Machine learning algorithms are viable alternatives to the traditional diagnoses of heart disease from ECG signals. However, the black box nature of complex machine learning algorithms and the difficulty in explaining a model's outcomes are obstacles for medical practitioners in having confidence in machine learning models. This observation paves the way for interpretable machine learning (IML) models as diagnostic tools that can build a physician's trust and provide evidence-based diagnoses. Therefore, in this systematic literature review, we studied and analyzed the research landscape in interpretable machine learning techniques by focusing on heart disease diagnosis from an ECG signal. In this regard, the contribution of our work is manifold; first, we present an elaborate discussion on interpretable machine learning techniques. In addition, we identify and characterize ECG signal recording datasets that are readily available for machine learning-based tasks. Furthermore, we identify the progress that has been achieved in ECG signal interpretation using IML techniques. Finally, we discuss the limitations and challenges of IML techniques in interpreting ECG signals.

MDITRE: Scalable and Interpretable Machine Learning for Predicting Host Status from Temporal Microbiome Dynamics

Longitudinal microbiome data sets are being generated with increasing regularity, and there is broad recognition that these studies are critical for unlocking the mechanisms through which the microbiome impacts human health and disease. However, there is a dearth of computational tools for analyzing microbiome time-series data. To address this gap, we developed an open-source software package, Microbiome Differentiable Interpretable Temporal Rule Engine (MDITRE), which implements a new highly efficient method leveraging deep-learning technologies to derive human-interpretable rules that predict host status from longitudinal microbiome data. Using semi-synthetic and a large compendium of publicly available 16S rRNA amplicon and metagenomics sequencing data sets, we demonstrate that in almost all cases, MDITRE performs on par with or better than popular uninterpretable machine learning methods, and orders-of-magnitude faster than the prior interpretable technique. MDITRE also provides a graphical user interface, which we show through case studies can be used to derive biologically meaningful interpretations linking patterns of microbiome changes over time with host phenotypes. IMPORTANCE The human microbiome, or collection of microbes living on and within us, changes over time. Linking these changes to the status of the human host is crucial to understanding how the microbiome influences a variety of human diseases. Due to the large scale and complexity of microbiome data, computational methods are essential. Existing computational methods for linking changes in the microbiome to the status of the human host are either unable to scale to large and complex microbiome data sets or cannot produce human-interpretable outputs. We present a new computational method and software package that overcomes the limitations of previous methods, allowing researchers to analyze larger and more complex data sets while producing easily interpretable outputs. Our method has the potential to enable new insights into how changes in the microbiome over time maintain health or lead to disease in humans and facilitate the development of diagnostic tests based on the microbiome.

Detecting Asymptomatic Infections of Rice Bacterial Leaf Blight Using Hyperspectral Imaging and 3-Dimensional Convolutional Neural Network With Spectral Dilated Convolution

Rice is one of the most important food crops for human beings. Its total production ranks third in the grain crop output. Bacterial Leaf Blight (BLB), as one of the three major diseases of rice, occurs every year, posing a huge threat to rice production and safety. There is an asymptomatic period between the infection and the onset periods, and BLB will spread rapidly and widely under suitable conditions. Therefore, accurate detection of early asymptomatic BLB is very necessary. The purpose of this study was to test the feasibility of detecting early asymptomatic infection of the rice BLB disease based on hyperspectral imaging and Spectral Dilated Convolution 3-Dimensional Convolutional Neural Network (SDC-3DCNN). First, hyperspectral images were obtained from rice leaves infected with the BLB disease at the tillering stage. The spectrum was smoothed by the Savitzky-Golay (SG) method, and the wavelength between 450 and 950 nm was intercepted for analysis. Then Principal Component Analysis (PCA) and Random Forest (RF) were used to extract the feature information from the original spectra as inputs. The overall performance of the SDC-3DCNN model with different numbers of input features and different spectral dilated ratios was evaluated. Lastly, the saliency map visualization was used to explain the sensitivity of individual wavelengths. The results showed that the performance of the SDC-3DCNN model reached an accuracy of 95.4427% when the number of inputs is 50 characteristic wavelengths (extracted by RF) and the dilated ratio is set at 5. The saliency-sensitive wavelengths were identified in the range from 530 to 570 nm, which overlaps with the important wavelengths extracted by RF. According to our findings, combining hyperspectral imaging and deep learning can be a reliable approach for identifying early asymptomatic infection of the rice BLB disease, providing sufficient support for early warning and rice disease prevention.

Inclusion of social determinants of health improves sepsis readmission prediction models

OBJECTIVE

Sepsis has a high rate of 30-day unplanned readmissions. Predictive modeling has been suggested as a tool to identify high-risk patients. However, existing sepsis readmission models have low predictive value and most predictive factors in such models are not actionable.

MATERIALS AND METHODS

Data from patients enrolled in the AllofUs Research Program cohort from 35 hospitals were used to develop a multicenter validated sepsis-related unplanned readmission model that incorporates clinical and social determinants of health (SDH) to predict 30-day unplanned readmissions. Sepsis cases were identified using concepts represented in the Observational Medical Outcomes Partnership. The dataset included over 60 clinical/laboratory features and over 100 SDH features.

RESULTS

Incorporation of SDH factors into our model of clinical and demographic features improves model area under the receiver operating characteristic curve (AUC) significantly (from 0.75 to 0.80; P < .001). Model-agnostic interpretability techniques revealed demographics, economic stability, and delay in getting medical care as important SDH predictive features of unplanned hospital readmissions.

DISCUSSION

This work represents one of the largest studies of sepsis readmissions using objective clinical data to date (8935 septic index encounters). SDH are important to determine which sepsis patients are more likely to have an unplanned 30-day readmission. The AllofUS dataset provides granular data from a diverse set of individuals, making this model potentially more generalizable than prior models.

CONCLUSION

Use of SDH improves predictive performance of a model to identify which sepsis patients are at high risk of an unplanned 30-day readmission.

Multi-label classification of reduced-lead ECGs using an interpretable deep convolutional neural network

Objective. We propose a model that can perform multi-label classification on 26 cardiac abnormalities from reduced lead Electrocardiograms (ECGs) and interpret the model.Approach. PhysioNet/Computing in Cardiology (CinC) challenge 2021 datasets are used to train the model. All recordings shorter than 20 seconds are preprocessed by normalizing, resampling, and zero-padding. The frequency domains of the recordings are obtained by applying Fast Fourier Transform. The time domain and frequency domain of the signals are fed into two separate deep convolutional neural networks. The outputs of these networks are then concatenated and passed through a fully connected layer that outputs the probabilities of 26 classes. Data imbalance is addressed by using a threshold of 0.13 to the sigmoid output. The 2-lead model is tested under noise contamination based on the quality of the signal and interpreted using SHapley Additive exPlanations (SHAP).Main results. The proposed method obtained a challenge score of 0.55, 0.51, 0.56, 0.55, and 0.56, ranking 2nd, 5th, 3rd, 3rd, and 3rd out of 39 officially ranked teams on 12-lead, 6-lead, 4-lead, 3-lead, and 2-lead hidden test datasets, respectively, in the PhysioNet/CinC challenge 2021. The model performs well under noise contamination with mean F1 scores of 0.53, 0.56 and 0.56 for the excellent, barely acceptable and unacceptable signals respectively. Analysis of the SHAP values of the 2-lead model verifies the performance of the model while providing insight into labeling inconsistencies and reasons for the poor performance of the model in some classes.Significance. We have proposed a model that can accurately identify 26 cardiac abnormalities using reduced lead ECGs that performs comparably with 12-lead ECGs and interpreted the model behavior. We demonstrate that the proposed model using only the limb leads performs with accuracy comparable to that using all 12 leads.

Interpretable, not black-box, artificial intelligence should be used for embryo selection

Artificial intelligence (AI) techniques are starting to be used in IVF, in particular for selecting which embryos to transfer to the woman. AI has the potential to process complex data sets, to be better at identifying subtle but important patterns, and to be more objective than humans when evaluating embryos. However, a current review of the literature shows much work is still needed before AI can be ethically implemented for this purpose. No randomized controlled trials (RCTs) have been published, and the efficacy studies which exist demonstrate that algorithms can broadly differentiate well between 'good-' and 'poor-' quality embryos but not necessarily between embryos of similar quality, which is the actual clinical need. Almost universally, the AI models were opaque ('black-box') in that at least some part of the process was uninterpretable. This gives rise to a number of epistemic and ethical concerns, including problems with trust, the possibility of using algorithms that generalize poorly to different populations, adverse economic implications for IVF clinics, potential misrepresentation of patient values, broader societal implications, a responsibility gap in the case of poor selection choices and introduction of a more paternalistic decision-making process. Use of interpretable models, which are constrained so that a human can easily understand and explain them, could overcome these concerns. The contribution of AI to IVF is potentially significant, but we recommend that AI models used in this field should be interpretable, and rigorously evaluated with RCTs before implementation. We also recommend long-term follow-up of children born after AI for embryo selection, regulatory oversight for implementation, and public availability of data and code to enable research teams to independently reproduce and validate existing models.

IoT and Interpretable Machine Learning Based Framework for Disease Prediction in Pearl Millet

Decrease in crop yield and degradation in product quality due to plant diseases such as rust and blast in pearl millet is the cause of concern for farmers and the agriculture industry. The stipulation of expert advice for disease identification is also a challenge for the farmers. The traditional techniques adopted for plant disease detection require more human intervention, are unhandy for farmers, and have a high cost of deployment, operation, and maintenance. Therefore, there is a requirement for automating plant disease detection and classification. Deep learning and IoT-based solutions are proposed in the literature for plant disease detection and classification. However, there is a huge scope to develop low-cost systems by integrating these techniques for data collection, feature visualization, and disease detection. This research aims to develop the 'Automatic and Intelligent Data Collector and Classifier' framework by integrating IoT and deep learning. The framework automatically collects the imagery and parametric data from the pearl millet farmland at ICAR, Mysore, India. It automatically sends the collected data to the cloud server and the Raspberry Pi. The 'Custom-Net' model designed as a part of this research is deployed on the cloud server. It collaborates with the Raspberry Pi to precisely predict the blast and rust diseases in pearl millet. Moreover, the Grad-CAM is employed to visualize the features extracted by the 'Custom-Net'. Furthermore, the impact of transfer learning on the 'Custom-Net' and state-of-the-art models viz. Inception ResNet-V2, Inception-V3, ResNet-50, VGG-16, and VGG-19 is shown in this manuscript. Based on the experimental results, and features visualization by Grad-CAM, it is observed that the 'Custom-Net' extracts the relevant features and the transfer learning improves the extraction of relevant features. Additionally, the 'Custom-Net' model reports a classification accuracy of 98.78% that is equivalent to state-of-the-art models viz. Inception ResNet-V2, Inception-V3, ResNet-50, VGG-16, and VGG-19. Although the classification of 'Custom-Net' is comparable to state-of-the-art models, it is effective in reducing the training time by 86.67%. It makes the model more suitable for automating disease detection. This proves that the proposed model is effective in providing a low-cost and handy tool for farmers to improve crop yield and product quality.

Deep Learning of Explainable EEG Patterns as Dynamic Spatiotemporal Clusters and Rules in a Brain-Inspired Spiking Neural Network

The paper proposes a new method for deep learning and knowledge discovery in a brain-inspired Spiking Neural Networks (SNN) architecture that enhances the model’s explainability while learning from streaming spatiotemporal brain data (STBD) in an incremental and on-line mode of operation. This led to the extraction of spatiotemporal rules from SNN models that explain why a certain decision (output prediction) was made by the model. During the learning process, the SNN created dynamic neural clusters, captured as polygons, which evolved in time and continuously changed their size and shape. The dynamic patterns of the clusters were quantitatively analyzed to identify the important STBD features that correspond to the most activated brain regions. We studied the trend of dynamically created clusters and their spike-driven events that occur together in specific space and time. The research contributes to: (1) enhanced interpretability of SNN learning behavior through dynamic neural clustering; (2) feature selection and enhanced accuracy of classification; (3) spatiotemporal rules to support model explainability; and (4) a better understanding of the dynamics in STBD in terms of feature interaction. The clustering method was applied to a case study of Electroencephalogram (EEG) data, recorded from a healthy control group (n = 21) and opiate use (n = 18) subjects while they were performing a cognitive task. The SNN models of EEG demonstrated different trends of dynamic clusters across the groups. This suggested to select a group of marker EEG features and resulted in an improved accuracy of EEG classification to 92%, when compared with all-feature classification. During learning of EEG data, the areas of neurons in the SNN model that form adjacent clusters (corresponding to neighboring EEG channels) were detected as fuzzy boundaries that explain overlapping activity of brain regions for each group of subjects.

Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition

The combination of multimodal imaging and genomics provides a more comprehensive way for the study of mental illnesses and brain functions. Deep network-based data fusion models have been developed to capture their complex associations, resulting in improved diagnosis of diseases. However, deep learning models are often difficult to interpret, bringing about challenges for uncovering biological mechanisms using these models. In this work, we develop an interpretable multimodal fusion model to perform automated diagnosis and result interpretation simultaneously. We name it Grad-CAM guided convolutional collaborative learning (gCAM-CCL), which is achieved by combining intermediate feature maps with gradient-based weights. The gCAM-CCL model can generate interpretable activation maps to quantify pixel-level contributions of the input features. Moreover, the estimated activation maps are class-specific, which can therefore facilitate the identification of biomarkers underlying different groups. We validate the gCAM-CCL model on a brain imaging-genetic study, and demonstrate its applications to both the classification of cognitive function groups and the discovery of underlying biological mechanisms. Specifically, our analysis results suggest that during task-fMRI scans, several object recognition related regions of interests (ROIs) are activated followed by several downstream encoding ROIs. In addition, the high cognitive group may have stronger neurotransmission signaling while the low cognitive group may have problems in brain/neuron development due to genetic variations.

An Ensemble Learning Solution for Predictive Maintenance of Wind Turbines Main Bearing

A novel and innovative solution addressing wind turbines’ main bearing failure predictions using SCADA data is presented. This methodology enables to cut setup times and has more flexible requirements when compared to the current predictive algorithms. The proposed solution is entirely unsupervised as it does not require the labeling of data through work orders logs. Results of interpretable algorithms, which are tailored to capture specific aspects of main bearing failures, are merged into a combined health status indicator making use of Ensemble Learning principles. Based on multiple specialized indicators, the interpretability of the results is greater compared to black-box solutions that try to address the problem with a single complex algorithm. The proposed methodology has been tested on a dataset covering more than two year of operations from two onshore wind farms, counting a total of 84 turbines. All four main bearing failures are anticipated at least one month of time in advance. Combining individual indicators into a composed one proved effective with regard to all the tracked metrics. Accuracy of 95.1%, precision of 24.5% and F1 score of 38.5% are obtained averaging the values across the two windfarms. The encouraging results, the unsupervised nature and the flexibility and scalability of the proposed solution are appealing, making it particularly attractive for any online monitoring system used on single wind farms as well as entire wind turbine fleets.