Quantification of errors in ordinal outcome scales using shannon entropy: effect on sample size calculations

The use of cloud based machine learning to predict outcome in intracerebral haemorrhage without explicit programming expertise

Machine Learning (ML) techniques require novel computer programming skills along with clinical domain knowledge to produce a useful model. We demonstrate the use of a cloud-based ML tool that does not require any programming expertise to develop, validate and deploy a prognostic model for Intracerebral Haemorrhage (ICH). The data of patients admitted with Spontaneous Intracerebral haemorrhage from January 2015 to December 2019 was accessed from our prospectively maintained hospital stroke registry. 80% of the dataset was used for training, 10% for validation, and 10% for testing. Seventeen input variables were used to predict the dichotomized outcomes (Good outcome mRS 0-3/ Bad outcome mRS 4-6), using machine learning (ML) and logistic regression (LR) models. The two different approaches were evaluated using Area Under the Curve (AUC) for Receiver Operating Characteristic (ROC), Precision recall and accuracy. Our data set comprised of a cohort of 1000 patients. The data was split 8:1 for training & testing respectively. The AUC ROC of the ML model was 0.86 with an accuracy of 75.7%. With LR AUC ROC was 0.74 with an accuracy of 73.8%. Feature importance chart showed that Glasgow coma score (GCS) at presentation had the highest relative importance, followed by hematoma volume and age in both approaches. Machine learning models perform better when compared to logistic regression. Models can be developed by clinicians possessing domain expertise and no programming experience using cloud based tools. The models so developed lend themselves to be incorporated into clinical workflow.

Radiomic-based nonlinear supervised learning classifiers on non-contrast CT to predict functional prognosis in patients with spontaneous intracerebral hematoma

PURPOSE

To evaluate if nonlinear supervised learning classifiers based on non-contrast CT can predict functional prognosis at discharge in patients with spontaneous intracerebral hematoma.

METHODS

Retrospective, single-center, observational analysis of patients with a diagnosis of spontaneous intracerebral hematoma confirmed by non-contrast CT between January 2016 and April 2018. Patients with HIE > 18 years and with TCCSC performed within the first 24 h of symptom onset were included. Patients with secondary spontaneous intracerebral hematoma and in whom radiomic variables were not available were excluded. Clinical, demographic and admission variables were collected. Patients were classified according to the Modified Rankin Scale (mRS) at discharge into good (mRS 0-2) and poor prognosis (mRS 3-6). After manual segmentation of each spontaneous intracerebral hematoma, the radiomics variables were obtained. The sample was divided into a training and testing cohort and a validation cohort (70-30% respectively). Different methods of variable selection and dimensionality reduction were used, and different algorithms were used for model construction. Stratified 10-fold cross-validation were performed on the training and testing cohort and the mean area under the curve (AUC) were calculated. Once the models were trained, the sensitivity of each was calculated to predict functional prognosis at discharge in the validation cohort.

RESULTS

105 patients with spontaneous intracerebral hematoma were analyzed. 105 radiomic variables were evaluated for each patient. P-SVM, KNN-E and RF-10 algorithms, in combination with the ANOVA variable selection method, were the best performing classifiers in the training and testing cohort (AUC 0.798, 0.752 and 0.742 respectively). The predictions of these models, in the validation cohort, had a sensitivity of 0.897 (0.778-1;95%CI), with a false-negative rate of 0% for predicting poor functional prognosis at discharge.

CONCLUSION

The use of radiomics-based nonlinear supervised learning classifiers are a promising diagnostic tool for predicting functional outcome at discharge in HIE patients, with a low false negative rate, although larger and balanced samples are still needed to develop and improve their performance.

Comparing health-related quality of life in modified Rankin Scale grades: 15D results from 323 patients with brain arteriovenous malformation and population controls

Background

We wanted to understand how patients with different modified Rankin Scale (mRS) grades differ regarding their health-related quality of life (HRQoL) and how this affects the interpretation and dichotomization of the grade.

Methods

In 2016, all adult patients in our brain arteriovenous malformation (AVM) database (n = 432) were asked to fill in mailed letters including a questionnaire about self-sufficiency and lifestyle and the 15D HRQoL questionnaire. The follow-up mRS was defined in 2016 using the electronic patient registry and the questionnaire data. The 15D profiles of each mRS grade were compared to those of the general population and to each other, using ANCOVA with age and sex standardization.

Results

Patients in mRS 0 (mean 15D score = 0.954 ± 0.060) had significantly better HRQoL than the general population (mean = 0.927 ± 0.028), p < 0.0001, whereas patients in mRS 1–4 had worse HRQoL than the general population, p < 0.0001. Patients in mRS 1 (mean = 0.844 ± 0.100) and mRS 2 (mean = 0.838 ± 0.107) had a similar HRQoL. In the recently published AVM research, the most commonly used cut points for mRS dichotomization were between mRS 1 and 2 and between mRS 2 and 3.

Conclusions

Using 15D, we were able to find significant differences in the HRQoL between mRS 0 and mRS 1 AVM patients, against the recent findings on stroke patients using EQ-5D in their analyses. Although the dichotomization cut point is commonly set between mRS 1 and 2, patients in these grades had a similar HRQoL and a decreased ability to continue their premorbid lifestyle, in contrast to patients in mRS 0.

Pooled analysis suggests benefit of catheter-based hematoma removal for intracerebral hemorrhage

OBJECTIVE

To develop models of outcome for intracerebral hemorrhage (ICH) to identify promising and futile interventions based on their early phase results without need for correction for baseline imbalances.

METHODS

We developed a pooled outcome model from the control arms of randomized control trials and tested different interventions against the model at comparable baseline conditions. Eligible clinical trials and large case series were identified from multiple library databases. Models based on baseline factors reported in the control arms were tested for the ability to predict functional outcome (modified Rankin Scale score) and mortality. Interventions were grouped into blood pressure control, fibrinolytic-assisted hematoma evacuation, hemostatic medications, and neuroprotective agents. Statistical intervals around the model were generated at the p = 0.1 level to screen how each trial's outcome compared to expected outcome.

RESULTS

Fourteen control arms with 3,386 patients were used to develop 7 alternate models for functional outcome. The model incorporating baseline NIH Stroke Scale, age, and hematoma volume yielded the best fit (adjusted R ² = 0.89). All early phase treatments that eventually resulted in negative late phase trials were identified as negative by this method. Early phase fibrinolytic-assisted hematoma evacuation studies showed the most promise trending toward improved functional outcome with no suggestion of an increase in mortality, supporting its further study.

CONCLUSIONS

We successfully developed an outcome model for ICH that identified interventions destined to be negative while identifying a promising one. Such an approach may assist in prioritizing resources prior to multicenter trial.

Functional Assessment for Acute Stroke Trials: Properties, Analysis, and Application

A measure of treatment effect is needed to assess the utility of any novel intervention in acute stroke. For a potentially disabling condition such as stroke, outcomes of interest should include some measure of functional recovery. There are many functional outcome assessments that can be used after stroke. In this narrative review, we discuss exemplars of assessments that describe impairment, activity, participation, and quality of life. We will consider the psychometric properties of assessment scales in the context of stroke trials, focusing on validity, reliability, responsiveness, and feasibility. We will consider approaches to the analysis of functional outcome measures, including novel statistical approaches. Finally, we will discuss how advances in audiovisual and information technology could further improve outcome assessment in trials.

Embracing Biological and Methodological Variance in a New Approach to Pre-Clinical Stroke Testing

High-profile failures in stroke clinical trials have discouraged clinical translation of neuroprotectants. While there are several plausible explanations for these failures, we believe that the fundamental problem is the way clinical and pre-clinical studies are designed and analyzed for heterogeneous disorders such as stroke due to innate biological and methodological variability that current methods cannot capture. Recent efforts to address pre-clinical rigor and design, while important, are unable to account for variability present even in genetically homogenous rodents. Indeed, efforts to minimize variability may lessen the clinical relevance of pre-clinical models. We propose a new approach that recognizes the important role of baseline stroke severity and other factors in influencing outcome. Analogous to clinical trials, we propose reporting baseline factors that influence outcome and then adapting for the pre-clinical setting a method developed for clinical trial analysis where the influence of baseline factors is mathematically modeled and the variance quantified. A new therapy's effectiveness is then evaluated relative to the pooled outcome variance at its own baseline conditions. In this way, an objective threshold for robustness can be established that must be overcome to suggest its effectiveness when expanded to broader populations outside of the controlled environment of the PI's laboratory. The method is model neutral and subsumes sources of variance as reflected in baseline factors such as initial stroke severity. We propose that this new approach deserves consideration for providing an objective method to select agents worthy of the commitment of time and resources in translation to clinical trials.

An Outcome Model for Intravenous rt-PA in Acute Ischemic Stroke

Most early phase trials in stroke and brain trauma have failed in phase 3, including efforts to improve acute ischemic stroke outcomes beyond that achieved by intravenous recombinant tissue plasminogen activator (t-PA) (IVT). With the exception of more recent stent retriever trials, most subsequent phase 3 trials failed. We previously showed that baseline imbalances, non-linear relationships of these factors to outcome, and unrepresentative control populations invalidate traditional statistical analysis in early trials of heterogeneous diseases such as stroke. We developed an alternative approach using a pooled outcome model derived from control arms of randomized clinical trial (RCTs). This model then permits comparing treatment trials to an expected outcome of a pooled population. Here, we hypothesized we could develop such a model for IVT and tested it against outcomes without IVT. We surveyed literature for all trials involving one arm with IVT reporting baseline National Institute Stroke Scale (NIHSS), age, and outcome. A non-linear fit was performed including multi-dimensional statistical intervals (±95 %) permitting visual comparison of outcomes at their own baselines. We compared models derived from non-IVT control arms. Models from 24 IVT RCTs representing 3195 subjects were successfully generated for functional outcome, modified Rankin Scale (mRS) 0-2 (r(2) = 0. 83, p < 0.001), and mortality (r(2) = 0.54; p = 0.001). We confirmed better outcomes compared to no IVT and mixed use IVT models across the range of baseline factors. It was possible to generate an expected outcome model for IVT from existing literature. We confirmed benefit compared to placebo. This model should be useful to compare to new agents without the need for statistical manipulation.

Improving early clinical trial phase identification of promising therapeutics

This review addresses decision-making underlying the frequent failure to confirm early-phase positive trial results and how to prioritize which early agents to transition to late phase. While unexpected toxicity is sometimes responsible for late-phase failures, lack of efficacy is also frequently found. In stroke as in other conditions, early trials often demonstrate imbalances in factors influencing outcome. Other issues complicate early trial analysis, including unequally distributed noise inherent in outcome measures and variations in natural history among studies. We contend that statistical approaches to correct for imbalances and noise, while likely valid for homogeneous conditions, appear unable to accommodate disease complexity and have failed to correctly identify effective agents. While blinding and randomization are important to reduce selection bias, these methods appear insufficient to insure valid conclusions. We found potential sources of analytical errors in nearly 90% of a sample of early stroke trials. To address these issues, we recommend changes in early-phase analysis and reporting: (1) restrict use of statistical correction to studies where the underlying assumptions are validated, (2) select dichotomous over continuous outcomes for small samples, (3) consider pooled samples to model natural history to detect early therapeutic signals and increase the likelihood of replication in larger samples, (4) report subgroup baseline conditions, (5) consider post hoc methods to restrict analysis to subjects with an appropriate match, and (6) increase the strength of effect threshold given these cumulative sources of noise and potential errors. More attention to these issues should lead to better decision-making regarding selection of agents to proceed to pivotal trials.

Is visual evaluation of aneurysm coiling a reliable study end point? Systematic review and meta-analysis

BACKGROUND AND PURPOSE

Angiographic occlusion as a surrogate marker of satisfactory aneurysm treatment is commonly used in clinical trials although some pitfalls have to be considered. To investigate the inter-rater reliability of visual rating of aneurysm occlusion as study end point, we performed a systematic review and meta-analysis.

METHODS

Electronic databases (MEDLINE, EMBASE, PubMed, and the Cochrane Library) were searched up to June 2014. Assessment of risk for bias was based on the Quality Appraisal Tool for Studies of Diagnostic Reliability and the Guidelines for Reporting Reliability and Agreement studies. Inter-rater reliability estimates were pooled across studies using meta-analysis, and the influence of several factors (eg, imaging methods, grading scales, and occlusion rate) was tested with meta-regression.

RESULTS

From 1193 titles, 644 abstracts and 87 full-text versions were reviewed. Twenty-six articles met the inclusion criteria and provided 77 reliability estimates. Twenty-one different rating scales were used, and statistical analysis varied. Mean inter-rater agreement of the pooled studies was substantial (κ=0.65; 95% confidence interval, 0.60-0.69). Reliability varied significantly as a function of imaging methods, grading scales, occlusion rates, and their interaction. Observer agreement substantially increased with increasing occlusion rate in digital subtraction angiography but not in MR angiography. Reliability was higher in studies using 2- or 3-value grading scales than in studies with 4-value grading scales.

CONCLUSIONS

There is significant heterogeneity between studies evaluating the reliability of visual evaluation of aneurysm coiling. On the basis of our analysis, we found that the combination of magnetic resonance angiography, 3-value grading scale, and 2 trained raters seems most promising for usage as surrogate study end points.

CHInese Medicine NeuroAiD Efficacy on Stroke Recovery - Extension Study (CHIMES-E): A Multicenter Study of Long-Term Efficacy

BACKGROUND

The CHInese Medicine NeuroAiD Efficacy on Stroke recovery (CHIMES) study was an international randomized double-blind placebo-controlled trial of MLC601 (NeuroAiD) in subjects with cerebral infarction of intermediate severity within 72 h. CHIMES-E (Extension) aimed at evaluating the effects of the initial 3-month treatment with MLC601 on long-term outcome for up to 2 years.

METHODS

All subjects randomized in CHIMES were eligible for CHIMES-E. Inclusion criteria for CHIMES were age ≥18, baseline National Institute of Health Stroke Scale of 6-14, and pre-stroke modified Rankin Scale (mRS) ≤1. Initial CHIMES treatment allocation blinding was maintained, although no further study treatment was provided in CHIMES-E. Subjects received standard care and rehabilitation as prescribed by the treating physician. mRS, Barthel Index (BI), and occurrence of medical events were ascertained at months 6, 12, 18, and 24. The primary outcome was mRS at 24 months. Secondary outcomes were mRS and BI at other time points.

RESULTS

CHIMES-E included 880 subjects (mean age 61.8 ± 11.3; 36% women). Adjusted OR for mRS ordinal analysis was 1.08 (95% CI 0.85-1.37, p = 0.543) and mRS dichotomy ≤1 was 1.29 (95% CI 0.96-1.74, p = 0.093) at 24 months. However, the treatment effect was significantly in favor of MLC601 for mRS dichotomy ≤1 at 6 months (OR 1.49, 95% CI 1.11-2.01, p = 0.008), 12 months (OR 1.41, 95% CI 1.05-1.90, p = 0.023), and 18 months (OR 1.36, 95% CI 1.01-1.83, p = 0.045), and for BI dichotomy ≥95 at 6 months (OR 1.55, 95% CI 1.14-2.10, p = 0.005) but not at other time points. Subgroup analyses showed no treatment heterogeneity. Rates of death and occurrence of vascular and other medical events were similar between groups.

CONCLUSIONS

While the benefits of a 3-month treatment with MLC601 did not reach statistical significance for the primary endpoint at 2 years, the odds of functional independence defined as mRS ≤1 was significantly increased at 6 months and persisted up to 18 months after a stroke.

Exploratory analysis of glyburide as a novel therapy for preventing brain swelling

BACKGROUND

Malignant infarction is characterized by the formation of cerebral edema, and medical treatment is limited. Preclinical data suggest that glyburide, an inhibitor of SUR1-TRPM4, is effective in preventing edema. We previously reported feasibility of the GAMES-Pilot study, a two-center prospective, open label, phase IIa trial of 10 subjects at high risk for malignant infarction based on diffusion weighted imaging (DWI) threshold of 82 cm(3) treated with RP-1127 (glyburide for injection). In this secondary analysis, we tested the hypothesis that RP-1127 may be efficacious in preventing poor outcome when compared to controls.

METHODS

Controls suffering large hemispheric infarction were obtained from the EPITHET and MMI-MRI studies. We first screened subjects for controls with the same DWI threshold used for enrollment into GAMES-Pilot, 82 cm(3). Next, to address imbalances, we applied a weighted Euclidean matching. Ninety day mRS 0-4, rate of decompressive craniectomy, and mortality were the primary clinical outcomes of interest.

RESULTS

The mean age of the GAMES cohort was 51 years and initial DWI volume was 102 ± 23 cm(3). After Euclidean matching, GAMES subjects showed similar NIHSS, higher DWI volume, younger age and had mRS 0-4-90% versus 50% in controls p = 0.049; with a similar trend in mRS 0-3 (40 vs. 25%; p = 0.43) and trend toward lower mortality (10 vs. 35%; p = 0.21).

CONCLUSIONS

In this pilot study, RP-1127-treated subjects showed better clinical outcomes when compared to historical controls. An adequately powered and randomized phase II trial of patients at risk for malignant infarction is needed to evaluate the potential efficacy of RP-1127.