Utility-weighted modified Rankin Scale: Still too crude to be a truly patient-centric primary outcome measure?

Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis

Background

Artificial intelligence-derived software technologies have been developed that are intended to facilitate the review of computed tomography brain scans in patients with suspected stroke.

Objectives

To evaluate the clinical and cost-effectiveness of using artificial intelligence-derived software to support review of computed tomography brain scans in acute stroke in the National Health Service setting.

Methods

Twenty-five databases were searched to July 2021. The review process included measures to minimise error and bias. Results were summarised by research question, artificial intelligence-derived software technology and study type. The health economic analysis focused on the addition of artificial intelligence-derived software-assisted review of computed tomography angiography brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke. The de novo model (developed in R Shiny, R Foundation for Statistical Computing, Vienna, Austria) consisted of a decision tree (short-term) and a state transition model (long-term) to calculate the mean expected costs and quality-adjusted life-years for people with ischaemic stroke and suspected large-vessel occlusion comparing artificial intelligence-derived software-assisted review to usual care.

Results

A total of 22 studies (30 publications) were included in the review; 18/22 studies concerned artificial intelligence-derived software for the interpretation of computed tomography angiography to detect large-vessel occlusion. No study evaluated an artificial intelligence-derived software technology used as specified in the inclusion criteria for this assessment. For artificial intelligence-derived software technology alone, sensitivity and specificity estimates for proximal anterior circulation large-vessel occlusion were 95.4% (95% confidence interval 92.7% to 97.1%) and 79.4% (95% confidence interval 75.8% to 82.6%) for Rapid (iSchemaView, Menlo Park, CA, USA) computed tomography angiography, 91.2% (95% confidence interval 77.0% to 97.0%) and 85.0 (95% confidence interval 64.0% to 94.8%) for Viz LVO (Viz.ai, Inc., San Fransisco, VA, USA) large-vessel occlusion, 83.8% (95% confidence interval 77.3% to 88.7%) and 95.7% (95% confidence interval 91.0% to 98.0%) for Brainomix (Brainomix Ltd, Oxford, UK) e-computed tomography angiography and 98.1% (95% confidence interval 94.5% to 99.3%) and 98.2% (95% confidence interval 95.5% to 99.3%) for Avicenna CINA (Avicenna AI, La Ciotat, France) large-vessel occlusion, based on one study each. These studies were not considered appropriate to inform cost-effectiveness modelling but formed the basis by which the accuracy of artificial intelligence plus human reader could be elicited by expert opinion. Probabilistic analyses based on the expert elicitation to inform the sensitivity of the diagnostic pathway indicated that the addition of artificial intelligence to detect large-vessel occlusion is potentially more effective (quality-adjusted life-year gain of 0.003), more costly (increased costs of £8.61) and cost-effective for willingness-to-pay thresholds of £3380 per quality-adjusted life-year and higher.

Limitations and conclusions

The available evidence is not suitable to determine the clinical effectiveness of using artificial intelligence-derived software to support the review of computed tomography brain scans in acute stroke. The economic analyses did not provide evidence to prefer the artificial intelligence-derived software strategy over current clinical practice. However, results indicated that if the addition of artificial intelligence-derived software-assisted review for guiding mechanical thrombectomy treatment decisions increased the sensitivity of the diagnostic pathway (i.e. reduced the proportion of undetected large-vessel occlusions), this may be considered cost-effective.

Future work

Large, preferably multicentre, studies are needed (for all artificial intelligence-derived software technologies) that evaluate these technologies as they would be implemented in clinical practice.

Study registration

This study is registered as PROSPERO CRD42021269609.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Evidence Synthesis programme (NIHR award ref: NIHR133836) and is published in full in Health Technology Assessment; Vol. 28, No. 11. See the NIHR Funding and Awards website for further award information.

Staged use of ordinal and linear disability scales: a practical approach to granular assessment of acute stroke outcome

Background

The modified Rankin Scale (mRS) assessment of global disability is the most common primary endpoint in acute stroke trials but lacks granularity (7 broad levels) and is ordinal (scale levels unknown distances apart), which constrains study power. Disability scales that are linear and continuous may better discriminate outcomes, but computerized administration in stroke patients is challenging. We, therefore, undertook to develop a staged use of an ordinal followed by a linear scale practical to use in multicenter trials.

Methods

Consecutive patients undergoing 3-month final visits in the NIH FAST-MAG phase 3 trial were assessed with the mRS followed by 15 mRS level-specific yes-no items of the Academic Medical Center Linear Disability Score (ALDS), a linear disability scale derived using item response theory.

Results

Among 55 patients, aged 71.2 (SD ± 14.2), 67% were men and the entry NIHSS was 10.7 (SD ± 9.5). At 90 days, the median mRS score was 3 (IQR, 1-4), and the median ALDS score was 78.8 (IQR, 3.3-100). ALDS scores correlated strongly with 90 days outcome measures, including the Barthel Index (r = 0.92), NIHSS (r = 0.87), and mRS (r = 0.94). ALDS scores also correlated modestly with entry NIHSS (r = 0.38). At 90 days, the ALDS showed greater scale granularity than the mRS, with fewer patients with identical values, 1.9 (SD ± 3.2) vs. 8.0 (SD ± 3.6), p < 0.001. When treatment effect magnitudes were small to moderate, projected trial sample size requirements were 2-12-fold lower when the ALDS rather than the mRS was used as the primary trial endpoint.

Conclusion

Among patients enrolled in an acute neuroprotective stroke trial, the ALDS showed strong convergent validity and superior discrimination characteristics compared with the modified Rankin Scale and increased projected trial power to detect clinically meaningful treatment benefits.

Generalised pairwise comparisons for trend: An extension to the win ratio and win odds for dose-response and prognostic variable analysis with arbitrary statements of outcome preference

The win ratio is a novel approach for handling complex patient outcomes that have seen considerable interest in the medical statistics literature, and operates by considering all-to-all pairwise statements of preference on outcomes. Recent extensions to the method have focused on the two-group case, with few developments made for considering the impact of a well-ordered explanatory variable, which would allow for dose-response analysis or the analysis of links between complex patient outcomes and prognostic variables. Where such methods have been developed, they are semiparametric methods that can only be applied to survival outcomes. In this article, we introduce the generalised pairwise comparison for trend, a modified form of Agresti's generalised odds ratio. This approach is capable of considering arbitrary statements of preference, thus enabling its use across all types of outcome data. We provide a simulation study validating the approach and illustrate it with three clinical applications in stroke research.

Improving Visualization Methods of Utility-Weighted Disability Outcomes for Stroke Trials

Background

The modified Rankin Scale (mRS) is the most common endpoint in acute stroke trials, but its power is limited when analyzed dichotomously and its indication of effect size is challenging to interpret when analyzed ordinally. To address these issues, the utility-weighted-mRS (UW-mRS) has been developed as a patient-centered, linear scale. However, appropriate data visualizations of UW-mRS results are needed, as current stacked bar chart displays do not convey crucial utility-weighting information.

Design/Methods

Two UW-mRS display formats were devised: (1) Utility Staircase charts, and (2) choropleth-stacked-bar-charts (CSBCs). In Utility Staircase displays, mRS segment height reflects the utility value of each mRS level. In CSBCs, mRS segment color intensity reflects the utility of each mRS level. Utility Staircase and CSBC figures were generated for 15 randomized comparisons of acute ischemic/hemorrhagic stroke therapies, including fibrinolysis, endovascular reperfusion, blood pressure moderation, and hemicraniectomy. Display accuracy in showing utility outcomes was assessed with the Tufte-lie-factor and ease-of-use assessed by formal ratings completed by a panel of 4 neurologists and emergency physicians and one nurse-coordinator.

Results

The Utility Staircase and CSBC displays rapidly conveyed patient-centered valuation of trial outcome distributions not available in conventional ordinal stacked bar charts. Tufte-lie-factor (LF) scores indicated “substantial distortion” of utility-valued outcomes for 93% (14/15) of conventional stacked bar charts, vs. “no distortion” for all Utility Staircase and CSBC displays. Clinician ratings on the Figural Display Questionnaire indicated that utility information encoded in row height (Utility Staircase display) was more readily assimilated than that conveyed in segment hue intensity (CSBC), both superior to conventional stacked bar charts.

Conclusions

Utility Staircase displays are an efficient graphical format for conveying utility weighted–modified Rankin Scale primary endpoint results of acute stroke trials, and choropleth-stacked-bar-charts a good alternative. Both are more accurate in depicting quantitative, health-related quality of life results and preferred by clinician users for utility results visualization, compared with conventional stacked bar charts.

Cost-Consequence Analysis of Advanced Imaging in Acute Ischemic Stroke Care

Introduction: The purpose of this study was to illustrate the potential costs and health consequences of implementing advanced CT angiography and perfusion (CTAP) as the initial imaging in patients presenting with acute ischemic stroke (AIS) symptoms at a comprehensive stroke center (CSC).

Methods: A decision-simulation model based on the American Heart Association's recommendations for AIS care pathways was developed to assess imaging strategies for a 5-year period from the institutional perspective. The following strategies were compared: (1) advanced CTAP imaging: NCCT + CTA + CT perfusion at the time of presentation; (2) standard-of-care: non-contrast CT (NCCT) at the time of presentation, with CT angiography (CTA) ± CT perfusion only in select patients (initial imaging to exclude hemorrhage and extensive ischemia) for mechanical thrombectomy (MT) evaluation. Model parameters were defined with evidence-based data. Cost-consequence and sensitivity analyses were performed. The modified Rankin Scale (mRS) at 90 days was used as the outcome measure.

Results: The decision-simulation modeling revealed that adoption of the advanced CTAP imaging increased per-patient imaging costs by 1.19% ($9.28/$779.72), increased per-patient treatment costs by 33.25% ($729.96/$2,195.24), and decreased other per-patient acute care costs by 0.7% (–$114.12/$16,285.85). The large increase in treatment costs was caused by higher proportion of patients being treated. However, improved outcomes lowered the other per-patient acute care costs. Over the five-year period, advanced CTAP imaging led to 1.63% (66/4,040) more patients with good outcomes (90-day mRS 0-2), 2.23% (66/2,960) fewer patients with poor outcomes (90-day mRS 3-5), and no change in mortality (90-day mRS 6). Our CT equipment utilization analysis showed that the demand for CT equipment in terms of scanner time (minutes) was 24% lower in the advanced CTAP imaging strategy compared to the standard-of-care strategy. The number of EVT procedures performed at the CSC may increase by 50%.

Conclusions: Our study reveals that adoption of advanced CTAP imaging at presentation increases the demand for treatment of acute ischemic stroke patients as more patients are diagnosed within the treatment time window compared to standard-of-care imaging. Advanced imaging also leads to more patients with good functional outcomes and fewer patients with dependent functional status.

Cost-Effectiveness Study of Initial Imaging Selection in Acute Ischemic Stroke Care

PURPOSE

National guidelines recommend prompt identification of candidates for acute ischemic stroke (AIS) treatment, requiring timely neuroimaging with CT and/or MRI. CT is often preferred because of its widespread availability and rapid acquisition. Despite higher diagnostic accuracy of MRI, it commonly involves complex workflows that could potentially cause treatment time delays. The purpose of this study was to analyze the impact on outcomes of imaging utilization before treatment decisions at comprehensive stroke centers for patients presenting with suspected AIS in the anterior circulation with last-known-well-to-arrival time 0 to 24 hours.

METHODS

A decision simulation model based on the American Heart Association's recommendations for AIS care pathways was developed from a health care perspective to compare initial imaging strategies: (1) stepwise-CT: noncontrast CT (NCCT) at the time of presentation, with CT angiography (CTA) ± CT perfusion (CTP) only in select patients (initial imaging to exclude hemorrhage and extensive ischemia) for mechanical thrombectomy (MT) evaluation; (2) stepwise-hybrid: NCCT at the time of presentation, with MR angiography (MRA) ± MR perfusion (MRP) only for MT evaluation; (3) stepwise-advanced: NCCT + CTA at presentation, with MR diffusion-weighted imaging (MR DWI) + MRP only for MT evaluation; (4) comprehensive-CT: NCCT + CTA + CTP at the time of presentation; and (5) comprehensive-MR: MR DWI + MRA + MRP at the time of presentation. Model parameters were defined using evidence-based data. Cost-effectiveness and sensitivity analyses were performed.

RESULTS

The cost-effectiveness analyses revealed that comprehensive-CT and comprehensive-MR yield the highest lifetime quality-adjusted life-years (QALYs) (4.81 and 4.82, respectively). However, the incremental cost-effectiveness ratio of comprehensive-MR is $233,000/QALY compared with comprehensive-CT. Stepwise-CT, stepwise-hybrid, and stepwise-advanced strategies are dominated, yielding lower QALYs and higher costs compared with comprehensive-CT.

CONCLUSIONS

Performing comprehensive-CT at presentation is the most cost-effective initial imaging strategy at comprehensive stroke centers.

Utility-Weighted Modified Rankin Scale Scores for the Assessment of Stroke Outcome: Pooled Analysis of 20 000+ Patients

BACKGROUND AND PURPOSE

Patient-centered care prioritizes patient beliefs and values towards wellbeing. We aimed to map functional status (modified Rankin Scale [mRS] scores) and health-related quality of life on the European Quality of Life 5-dimensional questionnaire (EQ-5D) to derive utility-weighted (UW) stroke outcome measures and test their statistical properties and construct validity.

METHODS

UW-mRS scores were derived using linear regression, with mRS as a discrete ordinal explanatory response variable in 8 large international acute stroke trials. Linear regression models were used to validate UW-mRS scores by assessing differences in mean UW-mRS scores between the treatment groups of each trial. To explore the variability in EQ-5D between individual mRS categories, we generated receiver operator characteristic curves for EQ-5D to differentiate between sequential mRS categories and misclassification matrix to classify individual patients into a matched mRS category based on the closest UW-mRS value to their observed individual EQ-5D value.

RESULTS

Among 22 946 acute stroke patients, derived UW-mRS across mRS scores 0 to 6 were 0.96, 0.83, 0.72, 0.54, 0.22, -0.18, and 0, respectively. Both UW-mRS and ordinal mRS scores captured divergent treatment effects across all 8 acute stroke trials. The sample sizes required to detect the treatment effects using UW-mRS scores as a continuous variable were almost half that required in trials for a binary cut point on the mRS. Area under receiver operator characteristic curves based on EQ-5D utility values varied from 0.66 to 0.81. Misclassification matrix showed moderate agreement between actual and matched mRS scores (kappa, 0.68 [95% CI, 0.67-0.68]).

CONCLUSIONS

Medical strategies that target avoiding dependency may provide maximum benefit in terms of poststroke health-related quality of life. Despite variable differences with mRS scores, the UW-mRS provides efficiency gains as a smaller sample size is required to detect a treatment effect in acute stroke trials through use of continuous scores. Registration: URL: https://www.clinicaltrials.gov; Unique identifiers: NCT00226096, NCT00716079, NCT01422616, NCT02162017, NCT00120003, NCT02123875. URL: http://ctri.nic.in; Unique identifier: CTRI/2013/04/003557. URL: https://www.isrctn.com; Unique identifier: ISRCTN89712435.

Weights for ordinal analyses of the modified Rankin Scale in stroke trials: A population-based cohort study

BACKGROUND

Ordinal/shift analyses of ordered measures like the modified Rankin Scale(mRS) are underused as primary trial outcomes for neurological disorders - despite statistical advantages - potentially hindered by poor clinical interpretability versus dichotomies, and by valuing state-transitions equally (linear scale). Weighted ordinal analyses incorporating step-changes at key transitions might have greater statistical validity and clinical applicability.

METHODS

In a prospective population-based cohort of ischaemic stroke (Oxford Vascular Study, recruited 2002-2014), we stratified 5-year outcomes of death, dementia, and/or institutionalization, health/social-care costs, and EuroQol-derived quality-adjusted life-expectancy(QALE) by 3-month mRS. We compared root-mean-square errors(RMSEs) from linear regressions for these outcomes with the mRS coded as a linear scale versus incorporating a spline at transitions 1-2, 2-3, or 3-4. We derived 3-month mRS weights for probability of 5-year death/dementia/institutionalization using age/sex-adjusted logistic regressions, and cost and QALE weights from 1000-bootstraps. We applied these weights to analyse recent trials of thrombectomy for acute ischaemic stroke.

FINDINGS

Among 1,607 patients, a non-linear (S-shaped) relationship was observed between 3-month mRS and each 5-year outcome, with RMSEs 18-73% lower using a spline at mRS 2-3 versus a linear representation. Age/sex-adjusted probability weights for 5-year death/dementia/institutionalization were: mRS 0=0.19; 1=0.27; 2=0.41; 3=0.73; 4=0.77; 5=0.94 (mRS 6=1 by definition). Similar trends were seen with costs; estimated 5-year QALEs were: mRS 0=3.88; 1=3.49; 2=3.01; 3=1.87; 4=1.30; 5=0.06; 6=0. Results were similar stratifying by age/sex, and excluding pre-morbidly disabled patients. Using a weighted ordinal approach, estimates of thrombectomy impact were more favourable than estimates with dichotomous approaches, 5-year cost reductions being 29% higher than with 0-2/3-6, and over three-fold higher than with 0-1/2-6 dichotomy.

INTERPRETATION

Our findings favour weighting the mRS in ordinal analyses for stroke and other neurological disorders, as state-transitions differ in clinical prognosis, quality-of-life, and costs. These weights could also be used for prognostication and cost-effectiveness analyses.

FUNDING

Wellcome Trust, Wolfson Foundation, NIHR Oxford Biomedical Research Centre, Rhodes Trust.

Health Utility Weighting of the Modified Rankin Scale: A Systematic Review and Meta-analysis

IMPORTANCE

The utility-weighted modified Rankin Scale (UW-mRS) has been proposed as a patient-centered alternative primary outcome for stroke clinical trials. However, to date, there is no clear consensus on an approach to weighting the mRS.

OBJECTIVE

To characterize the between-study variability in utility weighting of the mRS in a population of patients who experienced stroke and its implications when applied to the results of a clinical trial.

DATA SOURCES

In this systematic review and meta-analysis, MEDLINE, Embase, and PsycINFO were searched from January 1987 through May 2019 using major search terms for stroke, health utility, and mRS.

STUDY SELECTION

Original research articles published in English were reviewed. Included were studies with participants 18 years or older with ischemic or hemorrhagic stroke, transient ischemic attack, or subarachnoid hemorrhage, with mRS scores and utility weights evaluated concurrently. A total of 5725 unique articles were identified. Of these, 283 met criteria for full-text review, and 24 were included in the meta-analysis.

DATA EXTRACTION AND SYNTHESIS

PRISMA guidelines for systematic review were followed. Data extraction was performed independently by multiple researchers. Data were pooled using mixed models.

MAIN OUTCOMES AND MEASURES

The mean utility weights and 95% CIs were calculated for each mRS score and health utility scale. Geographic differences in weighting for the EuroQoL 5-dimension (EQ-5D) and Stroke Impact Scale-based UW-mRS were explored using inverse variance-weighted linear models. The results of 18 major acute stroke trials cited in current guidelines were then reanalyzed using the UW-mRS weighting scales identified in the systematic review.

RESULTS

The meta-analysis included 22 389 individuals; the mean (SD) age of participants was 65.9 (4.0) years, and the mean (SD) proportion of male participants was 58.2% (7.5%). For all health utility scales evaluated, statistically significant differences were observed between the mean utility weights by mRS score. For studies using an EQ-5D-weighted mRS, between-study variance was higher for worse (mRS 2-5) compared with better (mRS 0-1) scores. Of the 18 major acute stroke trials with reanalyzed results, 3 had an unstable outcome when using different UW-mRSs.

CONCLUSIONS AND RELEVANCE

Multiple factors, including cohort-specific characteristics and health utility scale selection, can influence mRS utility weighting. If the UW-mRS is selected as a primary outcome, the approach to weighting may alter the results of a clinical trial. Researchers using the UW-mRS should prospectively and concurrently obtain mRS scores and utility weights to characterize study-specific outcomes.