Performance of Automated Attenuation Measurements at Identifying Large Vessel Occlusion Stroke on CT Angiography

Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis

Background

Artificial intelligence-derived software technologies have been developed that are intended to facilitate the review of computed tomography brain scans in patients with suspected stroke.

Objectives

To evaluate the clinical and cost-effectiveness of using artificial intelligence-derived software to support review of computed tomography brain scans in acute stroke in the National Health Service setting.

Methods

Twenty-five databases were searched to July 2021. The review process included measures to minimise error and bias. Results were summarised by research question, artificial intelligence-derived software technology and study type. The health economic analysis focused on the addition of artificial intelligence-derived software-assisted review of computed tomography angiography brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke. The de novo model (developed in R Shiny, R Foundation for Statistical Computing, Vienna, Austria) consisted of a decision tree (short-term) and a state transition model (long-term) to calculate the mean expected costs and quality-adjusted life-years for people with ischaemic stroke and suspected large-vessel occlusion comparing artificial intelligence-derived software-assisted review to usual care.

Results

A total of 22 studies (30 publications) were included in the review; 18/22 studies concerned artificial intelligence-derived software for the interpretation of computed tomography angiography to detect large-vessel occlusion. No study evaluated an artificial intelligence-derived software technology used as specified in the inclusion criteria for this assessment. For artificial intelligence-derived software technology alone, sensitivity and specificity estimates for proximal anterior circulation large-vessel occlusion were 95.4% (95% confidence interval 92.7% to 97.1%) and 79.4% (95% confidence interval 75.8% to 82.6%) for Rapid (iSchemaView, Menlo Park, CA, USA) computed tomography angiography, 91.2% (95% confidence interval 77.0% to 97.0%) and 85.0 (95% confidence interval 64.0% to 94.8%) for Viz LVO (Viz.ai, Inc., San Fransisco, VA, USA) large-vessel occlusion, 83.8% (95% confidence interval 77.3% to 88.7%) and 95.7% (95% confidence interval 91.0% to 98.0%) for Brainomix (Brainomix Ltd, Oxford, UK) e-computed tomography angiography and 98.1% (95% confidence interval 94.5% to 99.3%) and 98.2% (95% confidence interval 95.5% to 99.3%) for Avicenna CINA (Avicenna AI, La Ciotat, France) large-vessel occlusion, based on one study each. These studies were not considered appropriate to inform cost-effectiveness modelling but formed the basis by which the accuracy of artificial intelligence plus human reader could be elicited by expert opinion. Probabilistic analyses based on the expert elicitation to inform the sensitivity of the diagnostic pathway indicated that the addition of artificial intelligence to detect large-vessel occlusion is potentially more effective (quality-adjusted life-year gain of 0.003), more costly (increased costs of £8.61) and cost-effective for willingness-to-pay thresholds of £3380 per quality-adjusted life-year and higher.

Limitations and conclusions

The available evidence is not suitable to determine the clinical effectiveness of using artificial intelligence-derived software to support the review of computed tomography brain scans in acute stroke. The economic analyses did not provide evidence to prefer the artificial intelligence-derived software strategy over current clinical practice. However, results indicated that if the addition of artificial intelligence-derived software-assisted review for guiding mechanical thrombectomy treatment decisions increased the sensitivity of the diagnostic pathway (i.e. reduced the proportion of undetected large-vessel occlusions), this may be considered cost-effective.

Future work

Large, preferably multicentre, studies are needed (for all artificial intelligence-derived software technologies) that evaluate these technologies as they would be implemented in clinical practice.

Study registration

This study is registered as PROSPERO CRD42021269609.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Evidence Synthesis programme (NIHR award ref: NIHR133836) and is published in full in Health Technology Assessment; Vol. 28, No. 11. See the NIHR Funding and Awards website for further award information.

FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval

Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.

Deep-learning based detection of vessel occlusions on CT-angiography in patients with suspected acute ischemic stroke

Swift diagnosis and treatment play a decisive role in the clinical outcome of patients with acute ischemic stroke (AIS), and computer-aided diagnosis (CAD) systems can accelerate the underlying diagnostic processes. Here, we developed an artificial neural network (ANN) which allows automated detection of abnormal vessel findings without any a-priori restrictions and in <2 minutes. Pseudo-prospective external validation was performed in consecutive patients with suspected AIS from 4 different hospitals during a 6-month timeframe and demonstrated high sensitivity (≥87%) and negative predictive value (≥93%). Benchmarking against two CE- and FDA-approved software solutions showed significantly higher performance for our ANN with improvements of 25-45% for sensitivity and 4-11% for NPV (p ≤ 0.003 each). We provide an imaging platform ( https://stroke.neuroAI-HD.org ) for online processing of medical imaging data with the developed ANN, including provisions for data crowdsourcing, which will allow continuous refinements and serve as a blueprint to build robust and generalizable AI algorithms.

The diagnostic performance of artificial intelligence algorithms for identifying M2 segment middle cerebral artery occlusions: A systematic review and meta-analysis

BACKGROUND

Artificial intelligence (AI)-based algorithms have been developed to facilitate rapid and accurate computed tomography angiography (CTA) assessment in proximal large vessel occlusion (LVO) acute ischemic stroke, including internal carotid artery and M1 occlusions. In clinical practice, however, the detection of medium vessel occlusion (MeVO) represents an ongoing diagnostic challenge in which the added value of AI remains unclear.

PURPOSE

To assess the diagnostic performance of AI platforms for detecting M2 occlusions.

METHODS

Studies that report the diagnostic performance of AI-based detection of M2 occlusions were screened, and sensitivity and specificity data were extracted using the semi-automated AutoLit software (Nested Knowledge, MN) platform. STATA (version 16 IC; Stata Corporation, College Station, Texas, USA) was used to conduct all analyses.

RESULTS

Eight studies with a low risk of bias and significant heterogeneity were included in the quantitative and qualitative synthesis. The pooled estimates of sensitivity and specificity of AI platforms for M2 occlusion detection were 64% (95% CI, 53 to 74%) and 97% (95% CI, 84 to 100%), respectively. The area under the curve (AUC) in the SROC curve was 0.79 (95% CI, 0.74 to 0.83).

CONCLUSION

The current performance of the AI-based algorithm makes it more suitable as an adjunctive confirmatory tool rather than as an independent one for M2 occlusions. With the rapid development of such algorithms, it is anticipated that newer generations will likely perform much better.

Emerging Detection Techniques for Large Vessel Occlusion Stroke: A Scoping Review

Background: Large vessel occlusion (LVO) is the obstruction of large, proximal cerebral arteries and can account for up to 46% of acute ischaemic stroke (AIS) when both the A2 and P2 segments are included (from the anterior and posterior cerebral arteries). It is of paramount importance that LVO is promptly recognised to provide timely and effective acute stroke management. This review aims to scope recent literature to identify new emerging detection techniques for LVO. As a good comparator throughout this review, the commonly used National Institutes of Health Stroke Scale (NIHSS), at a cut-off of ≥11, has been reported to have a sensitivity of 86% and a specificity of 60% for LVO. Methods: Four electronic databases (Medline via OVID, CINAHL, Scopus, and Web of Science), and grey literature using OpenGrey, were systematically searched for published literature investigating developments in detection methods for LVO, reported from 2015 to 2021. The protocol for the search was published with the Open Science Framework (10.17605/OSF.IO/A98KN). Two independent researchers screened the titles, abstracts, and full texts of the articles, assessing their eligibility for inclusion. Results: The search identified 5,082 articles, in which 2,265 articles were screened to assess their eligibility. Sixty-two studies remained following full-text screening. LVO detection techniques were categorised into 5 groups: stroke scales (n = 30), imaging and physiological methods (n = 15), algorithmic and machine learning approaches (n = 9), physical symptoms (n = 5), and biomarkers (n = 3). Conclusions: This scoping review has explored literature on novel and advancements in pre-existing detection methods for LVO. The results of this review highlight LVO detection techniques, such as stroke scales and biomarkers, with good sensitivity and specificity performance, whilst also showing advancements to support existing LVO confirmatory methods, such as neuroimaging.

Machine Learning Automated Detection of Large Vessel Occlusion From Mobile Stroke Unit Computed Tomography Angiography

BACKGROUND AND PURPOSE

Prehospital automated large vessel occlusion (LVO) detection in Mobile Stroke Units (MSUs) could accelerate identification and treatment of patients with LVO acute ischemic stroke. Here, we evaluate the performance of a machine learning (ML) model on CT angiograms (CTAs) obtained from 2 MSUs to detect LVO.

METHODS

Patients evaluated on MSUs in Houston and Los Angeles with out-of-hospital CTAs were identified. Anterior circulation LVO was defined as an occlusion of the intracranial internal carotid artery, middle cerebral artery (M1 or M2), or anterior cerebral artery vessels and determined by an expert human reader. A ML model to detect LVO was trained and tested on independent data sets consisting of in-hospital CTAs and then tested on MSU CTA images. Model performance was determined using area under the receiver-operator curve statistics.

RESULTS

Among 68 patients with out-of-hospital MSU CTAs, 40% had an LVO. The most common occlusion location was the middle cerebral artery M1 segment (59%), followed by the internal carotid artery (30%), and middle cerebral artery M2 (11%). Median time from last known well to CTA imaging was 88.0 (interquartile range, 59.5-196.0) minutes. After training on 870 in-hospital CTAs, the ML model performed well in identifying LVO in a separate in-hospital data set of 441 images with area under receiver-operator curve of 0.84 (95% CI, 0.80-0.87). ML algorithm analysis time was under 1 minute. The performance of the ML model on the MSU CTA images was comparable with area under receiver-operator curve 0.80 (95% CI, 0.71-0.89). There was no significant difference in performance between the Houston and Los Angeles MSU CTA cohorts.

CONCLUSIONS

In this study of patients evaluated on MSUs in 2 cities, a ML algorithm was able to accurately and rapidly detect LVO using prehospital CTA acquisitions.