Susceptibility of AutoML mortality prediction algorithms to model drift caused by the COVID pandemic

BACKGROUND

Concept drift and covariate shift lead to a degradation of machine learning (ML) models. The objective of our study was to characterize sudden data drift as caused by the COVID pandemic. Furthermore, we investigated the suitability of certain methods in model training to prevent model degradation caused by data drift.

METHODS

We trained different ML models with the H2O AutoML method on a dataset comprising 102,666 cases of surgical patients collected in the years 2014-2019 to predict postoperative mortality using preoperatively available data. Models applied were Generalized Linear Model with regularization, Default Random Forest, Gradient Boosting Machine, eXtreme Gradient Boosting, Deep Learning and Stacked Ensembles comprising all base models. Further, we modified the original models by applying three different methods when training on the original pre-pandemic dataset: (Rahmani K, et al, Int J Med Inform 173:104930, 2023) we weighted older data weaker, (Morger A, et al, Sci Rep 12:7244, 2022) used only the most recent data for model training and (Dilmegani C, 2023) performed a z-transformation of the numerical input parameters. Afterwards, we tested model performance on a pre-pandemic and an in-pandemic data set not used in the training process, and analysed common features.

RESULTS

The models produced showed excellent areas under receiver-operating characteristic and acceptable precision-recall curves when tested on a dataset from January-March 2020, but significant degradation when tested on a dataset collected in the first wave of the COVID pandemic from April-May 2020. When comparing the probability distributions of the input parameters, significant differences between pre-pandemic and in-pandemic data were found. The endpoint of our models, in-hospital mortality after surgery, did not differ significantly between pre- and in-pandemic data and was about 1% in each case. However, the models varied considerably in the composition of their input parameters. None of our applied modifications prevented a loss of performance, although very different models emerged from it, using a large variety of parameters.

CONCLUSIONS

Our results show that none of our tested easy-to-implement measures in model training can prevent deterioration in the case of sudden external events. Therefore, we conclude that, in the presence of concept drift and covariate shift, close monitoring and critical review of model predictions are necessary.

Enabling personalized perioperative risk prediction by using a machine-learning model based on preoperative data

Preoperative risk assessment is essential for shared decision-making and adequate perioperative care. Common scores provide limited predictive quality and lack personalized information. The aim of this study was to create an interpretable machine-learning-based model to assess the patient's individual risk of postoperative mortality based on preoperative data to allow analysis of personal risk factors. After ethical approval, a model for prediction of postoperative in-hospital mortality based on preoperative data of 66,846 patients undergoing elective non-cardiac surgery between June 2014 and March 2020 was created with extreme gradient boosting. Model performance and the most relevant parameters were shown using receiver operating characteristic (ROC-) and precision-recall (PR-) curves and importance plots. Individual risks of index patients were presented in waterfall diagrams. The model included 201 features and showed good predictive abilities with an area under receiver operating characteristic (AUROC) curve of 0.95 and an area under precision-recall curve (AUPRC) of 0.109. The feature with the highest information gain was the preoperative order for red packed cell concentrates followed by age and c-reactive protein. Individual risk factors could be identified on patient level. We created a highly accurate and interpretable machine learning model to preoperatively predict the risk of postoperative in-hospital mortality. The algorithm can be used to identify factors susceptible to preoperative optimization measures and to identify risk factors influencing individual patient risk.

Impact of the Covid-19 pandemic on the performance of machine learning algorithms for predicting perioperative mortality

BACKGROUND

Machine-learning models are susceptible to external influences which can result in performance deterioration. The aim of our study was to elucidate the impact of a sudden shift in covariates, like the one caused by the Covid-19 pandemic, on model performance.

METHODS

After ethical approval and registration in Clinical Trials (NCT04092933, initial release 17/09/2019), we developed different models for the prediction of perioperative mortality based on preoperative data: one for the pre-pandemic data period until March 2020, one including data before the pandemic and from the first wave until May 2020, and one that covers the complete period before and during the pandemic until October 2021. We applied XGBoost as well as a Deep Learning neural network (DL). Performance metrics of each model during the different pandemic phases were determined, and XGBoost models were analysed for changes in feature importance.

RESULTS

XGBoost and DL provided similar performance on the pre-pandemic data with respect to area under receiver operating characteristic (AUROC, 0.951 vs. 0.942) and area under precision-recall curve (AUPR, 0.144 vs. 0.187). Validation in patient cohorts of the different pandemic waves showed high fluctuations in performance from both AUROC and AUPR for DL, whereas the XGBoost models seemed more stable. Change in variable frequencies with onset of the pandemic were visible in age, ASA score, and the higher proportion of emergency operations, among others. Age consistently showed the highest information gain. Models based on pre-pandemic data performed worse during the first pandemic wave (AUROC 0.914 for XGBoost and DL) whereas models augmented with data from the first wave lacked performance after the first wave (AUROC 0.907 for XGBoost and 0.747 for DL). The deterioration was also visible in AUPR, which worsened by over 50% in both XGBoost and DL in the first phase after re-training.

CONCLUSIONS

A sudden shift in data impacts model performance. Re-training the model with updated data may cause degradation in predictive accuracy if the changes are only transient. Too early re-training should therefore be avoided, and close model surveillance is necessary.

Absence of a diurnal rhythm of oxytocin and arginine-vasopressin in human cerebrospinal fluid, blood and saliva

PURPOSE

The aims of our study were to determine first circadian influences on central concentrations of the neuropeptides oxytocin and arginine-vasopressin and second to investigate if these central concentrations are associated with those in the peripheral compartments blood and saliva in neurocritical care patients. We therefore included patients with external ventricular drain who attended a neurosurgical intensive care unit and were not exposed to painful or stressful stimuli during the sampling period. For this purpose, blood, cerebrospinal fluid and saliva were collected in a 24-hour-interval at the timepoints 06:00, 12:00, 18:00 and 24:00.

RESULTS

In none of the three body fluids examined, significant time-dependent fluctuations of oxytocin and arginine-vasopressin concentrations could be detected during the 24-hour sampling period. The only exception was the subgroup of postmenopausal women whose oxytocin concentrations in cerebrospinal fluid at 12:00 were significantly higher than at 18:00. Correlations of blood and cerebrospinal fluid and blood and saliva neuropeptide levels were very weak to weak at each timepoint. Cerebrospinal fluid and saliva oxytocin levels showed a moderate correlation at 06:00 but did correlate very weak at the other timepoints.

CONCLUSIONS

Central as well as peripheral oxytocin and arginine-vasopressin concentrations in neurocritical care patients did not show significant diurnal fluctuations. No strong correlations between central and peripheral neuropeptide concentrations could be detected under basal conditions. If investigators even though decide to use saliva concentrations as surrogate parameter for central neuropeptide activity, they have to consider that correlations of cerebrospinal fluid and saliva oxytocin seem to be highest in the early morning.

Influence of perioperative stress on central and peripheral oxytocin and arginine-vasopressin concentrations

Perioperative stress provides not only physical, but also psychic and emotional aspects, which may influence the hypothalamic neuropeptide system. Studies investigating the perioperative course of central neuropeptide activity are missing. Therefore, the present study aimed to determine perioperative fluctuations in central and concomitant peripheral concentrations of the hypothalamic neuropeptides oxytocin (OXT) and arginine-vasopressin (AVP), as well as their impact on perioperative anxiety and depression. Cerebrospinal fluid (CSF), blood and saliva were collected from 12 patients who underwent elective endovascular aortic repair with a routinely inserted spinal catheter. AVP and OXT concentrations were analysed at four timepoints: (i) the evening before the operation; (ii) the operation day immediately before anaesthesia induction; (iii) intraoperatively after the stent was placed; and (iv) on day 1 after the operation. Patients completed the Hospital Anxiety and Depression Scale (HADS) at timepoints 1 and 4. For CSF OXT, the present study showed a significant intraoperative decline, accompanied by a decrease in saliva. OXT blood concentrations before anaesthesia induction were higher than at the evening before the operation. OXT concentrations in CSF and saliva correlated well at timepoints 2-4. AVP concentrations in CSF, blood and saliva did not show any significant changes perioperatively. However, postoperative AVP blood concentrations showed a significant negative correlation with anxiety and depression scores according to the HADS. This pilot study demonstrates perioperative fluctuations in central OXT concentrations, which are better reflected by saliva than by blood. Further studies are required to determine whether OXT and AVP can predict postoperative post-traumatic stress disorder.

Neuropeptide Y - an early biomarker for cerebral vasospasm after aneurysmal subarachnoid hemorrhage

OBJECTIVES

In the human brain, the potent vasoconstrictive neuropeptide Y (NPY) is abundantly expressed. Neuropeptide Y, which is stored in perivascular nerve fibers of the cerebral arteries, regulates the cerebral vascular diameter as well as cerebral blood flow. However, the role of NPY in the pathogenesis of cerebral vasospasm (CV) related to subarachnoid hemorrhage (SAH) is unclear. We prospectively analyzed and compared the release of endogenous NPY in the cerebrospinal fluid (CSF) of 66 patients with SAH to NPY release in a control group. Additionally, we correlated the levels of NPY with CV and consecutive ischemic stroke.

METHODS

Sixty-six consecutive patients (40 women, 26 men; mean age 53·1 years) with aneurysmal SAH were included. In the SAH group, CSF was drawn daily from day 1 to day 10 after the onset of SAH. The CSF of 29 patients undergoing spinal anesthesia for orthopedic surgery served as control samples. The NPY levels were determined in duplicate CSF samples by means of a competitive enzyme immunoassay (EIA). The levels of NPY in CSF were correlated with the development of CV over the 10-day period after the onset of SAH and to the occurrence of consecutive ischemic stroke. To evaluate CSF NPY levels as a predictive biomarker for vasospasm, we calculated the sensitivity and specificity as well as the positive and negative predictive values.

RESULTS

The NPY levels were significantly higher in the SAH group than in the control group (p < 0·001). The treatment modality (clip versus coil) did not influence the level of NPY in CSF (p > 0·05). Patients with CV showed significantly higher NPY levels than patients without CV during the entire observation period. The NPY levels of the non-CV group dissipated over time, whereas the CV group showed continuously increasing values. The NPY levels from day 4 to 10 were significantly higher in patients with CV-related stroke than in non-stroke patients. Using 0·3 ng/ml as a cut-off value, NPY levels on day 3 predicted the occurrence of CV with a sensitivity and specificity of 82% and 72%, respectively. High NPY levels, starting on day 4, significantly correlated with poor Glasgow Outcome Score grading at the follow-up (p < 0·05).

DISCUSSION

Our data indicate that NPY is involved in the pathogenesis of SAH-related CV and ischemia. Neuropeptide Y represents an early and reliable biomarker for the prediction of CV and consecutive stroke due to aneurysmal SAH.

Pituitary dysfunction after aneurysmal subarachnoid hemorrhage

Patients recovering from aneurysmal SAH often complain about weakness, fatigue and impaired cognitive skills. Pituitary dysfunction might be one possible reason for these complaints, as in patients with traumatic brain injury, hypopituitarism is known to be a common complication. There are only a few studies dealing with this problem in SAH patients, but these studies suggest that pituitary disturbances are very frequent after aneurysmal SAH. But anterior pituitary lobe disturbances might not be the only one responsible for some complaints or complications in patients suffering from aneurysmal SAH. Hyponatremia in the early state after SAH could be a hint for posterior pituitary lobe dysfunction.

Plasma oxytocin and vasopressin do not predict neuropeptide concentrations in human cerebrospinal fluid

The involvement of the neuropeptides oxytocin (OXT) and vasopressin (AVP) in human socio-emotional behaviours is attracting increasing attention. There is ample evidence for elevated plasma levels upon a wide variety of social and emotional stimuli and scenarios, ranging from romantic love via marital distress up to psychopathology, with cause versus consequence being largely unclear. The present study examined whether plasma levels of both OXT and AVP are reflective of central neuropeptide levels, as assumed to impact upon socio-emotional behaviours. Concomitant plasma and cerebrospinal fluid (CSF) samples were taken from 41 non-neurological and nonpsychiatric patients under basal conditions. Although OXT and AVP levels in the CSF exceeded those in plasma, there was no correlation between both compartments, clearly suggesting that plasma OXT and AVP do not predict central neuropeptide concentrations. Thus, the validity of plasma OXT and AVP as potential biomarkers of human behaviour needs further clarification.

Soluble amyloid precursor protein β as blood-based biomarker of Alzheimer's disease

The aim of this study was to explore concentrations differences of soluble amyloid precursor protein (sAPP) α and β in blood plasma in patients with probable Alzheimer's disease (AD) and cognitively healthy age-matched control subjects, as well as patients with behavioural variant frontotemporal dementia (bvFTD). Concentrations of sAPPα and β were measured using enzyme-linked immunosorbent assay technology in 80 patients with probable AD, 37 age-matched control subjects and 14 patients with bvFTD. Concentration differences were explored using parametric tests. Significantly decreased plasma concentrations in the AD group compared with both the control group and the bvFTD group were detected for sAPPβ (P = 0.03 for both group comparisons), but not for sAPPα. The study provides a further piece of evidence in support of sAPPβ as a promising new biomarker of AD, which may potentially improve the diagnostic accuracy of existing markers and also enable a less invasive diagnostic workup. Further research is required to establish normal ranges and to replicate the results in independent cohorts including larger numbers of participants covering a wider spectrum of cognitive impairment.

Calcitonin-gene related peptide and cerebral vasospasm

The pathophysiology of arterial vasospasm following subarachnoid hemorrhage (SAH) is poorly understood and the contribution of endogenous neuropeptides has not been sufficiently elucidated. Recently, we detected an excessive release of vasoconstrictive neuropeptide Y (NPY) in SAH patients and identified a significant correlation of NPY cerebrospinal fluid (CSF) levels with vasospasm-related ischemia. Here, we present the results of an experimental study on the possible role of the potent endogenous vasodilator calcitonin-gene related peptide (CGRP) in the acute stage of SAH. Twelve consecutive patients with SAH were included. Seven patients had severe arterial vasospasm, confirmed by transcranial doppler-sonography (TCD). Prospectively, CSF was collected from day 1 to day 10 after onset of the SAH. The levels of CGRP were determined in a competitive enzyme immunoassay and were correlated with the clinical course and hemodynamic changes. A cohort of 29 patients without CNS disease served as a control. CGRP was significantly higher in SAH patients compared with the control group (p<0.05). From day 1 to day 4, the CGRP levels in patients without vasospasm were significantly higher than the levels of CGRP in patients with vasospasm (p<0.05). These patients did not develop cerebral ischemia. The significantly increased levels of the CGRP during the first days after onset of the SAH in the non-vasospasm group indicate a potential protective role of CGRP. CGRP may alleviate arterial vasoconstriction and thus protect the brain from vasospasm and subsequent ischemia.

The possible role of neuropeptide Y after spontaneous subarachnoid hemorrhage

OBJECTIVE

Neuropeptide Y (NPY), a highly potent vasoconstrictive neuropeptide, is widely expressed in the human brain, regulating vessel diameter and cerebral blood flow. Earlier studies focusing on the possible role of NPY in the context of aneurismal subarachnoid hemorrhage (SAH) and vasospasm have produced conflicting results. However, despite extensive research efforts, the pathophysiological mechanisms underlying the SAH-related vasospasm and delayed cerebral ischemia (DCI) have not been clarified. We, therefore, attempted to investigate the role of NPY in SAH-induced vasospasm in a larger, well documented patient population utilizing modern analytical tools. We focused on the release of the potent vasoconstrictor NPY in cerebrospinal fluid (CSF) and blood, and its correlation to vasospasm and stroke in the early clinical stage.

METHODS

Thirty-seven patients with SAH and a control group consisting of 29 patients were included. Eighteen patients developed stroke, 21 patients met the Doppler sonographical criteria for vasospasm. Twenty-nine patients had aneurysms of the anterior circulation and four patients of the posterior circulation. All patients had ventricular drainage inserted and an arterial catheter. Blood and CSF were drawn daily for NPY analysis during a 10-day interval.

RESULTS

The levels of NPY in CSF and plasma were significantly higher after SAH than in the control group (p = 0.001). The vasospasm group showed NPY levels in CSF which continuously ranged above the NPY levels of the non-vasospasm group (p = 0.001). Patients with stroke caused by vasospasm had significantly higher levels of NPY (p = 0.001).

DISCUSSION

NPY is released excessively into blood and CSF following SAH. Patients with cerebral infarction caused by vasospasm had significantly higher levels of NPY. Our results indicate a certain role for NPY in the pathophysiology of vasospasm due to SAH and justify further studies in this area of research.

Elevated premorbid body mass index is not associated with poor neurological outcome in the subacute state after aneurysmal subarachnoid hemorrhage

BACKGROUND

An elevated body mass index (BMI) is suggested to be a risk factor for a poor outcome after intracranial aneurysm rupture and is considered to be associated with cerebral infarction in patients with aneurysmal subarachnoid hemorrhage (SAH). The aim of this study was to analyze the association between permorbid BMI and neurological outcome.

METHODS

In this retrospective study, the patients' BMI at the time of their admission to hospital was correlated to their neurological outcome as measured by the Glasgow outcome score after two weeks and two months of treatment.

RESULTS

In contrast to other studies, there were no significant correlations between premorbid BMI and neurological outcome, shunt requirement, tracheotomy requirement and duration of stay on the intensive care unit (ICU).

CONCLUSIONS

Overweight patients have no higher risk of a poor neurological outcome after aneurysmal SAH if premorbid risk factors such as hypertension and hyperglycemia are carefully modified throughout the period of critical care.