Addressing the Challenges of Implementing Artificial Intelligence Tools in Clinical Practice: Principles From Experience

The multitude of artificial intelligence (AI)-based solutions, vendors, and platforms poses a challenging proposition to an already complex clinical radiology practice. Apart from assessing and ensuring acceptable local performance and workflow fit to improve imaging services, AI tools require multiple stakeholders, including clinical, technical, and financial, who collaborate to move potential deployable applications to full clinical deployment in a structured and efficient manner. Postdeployment monitoring and surveillance of such tools require an infrastructure that ensures proper and safe use. Herein, the authors describe their experience and framework for implementing and supporting the use of AI applications in radiology workflow.

Auto-Detection of Motion Artifacts on CT Pulmonary Angiograms with a Physician-Trained AI Algorithm

Purpose: Motion-impaired CT images can result in limited or suboptimal diagnostic interpretation (with missed or miscalled lesions) and patient recall. We trained and tested an artificial intelligence (AI) model for identifying substantial motion artifacts on CT pulmonary angiography (CTPA) that have a negative impact on diagnostic interpretation. Methods: With IRB approval and HIPAA compliance, we queried our multicenter radiology report database (mPower, Nuance) for CTPA reports between July 2015 and March 2022 for the following terms: "motion artifacts", "respiratory motion", "technically inadequate", and "suboptimal" or "limited exam". All CTPA reports were from two quaternary (Site A, n = 335; B, n = 259) and a community (C, n = 199) healthcare sites. A thoracic radiologist reviewed CT images of all positive hits for motion artifacts (present or absent) and their severity (no diagnostic effect or major diagnostic impairment). Coronal multiplanar images from 793 CTPA exams were de-identified and exported offline into an AI model building prototype (Cognex Vision Pro, Cognex Corporation) to train an AI model to perform two-class classification ("motion" or "no motion") with data from the three sites (70% training dataset, n = 554; 30% validation dataset, n = 239). Separately, data from Site A and Site C were used for training and validating; testing was performed on the Site B CTPA exams. A five-fold repeated cross-validation was performed to evaluate the model performance with accuracy and receiver operating characteristics analysis (ROC). Results: Among the CTPA images from 793 patients (mean age 63 ± 17 years; 391 males, 402 females), 372 had no motion artifacts, and 421 had substantial motion artifacts. The statistics for the average performance of the AI model after five-fold repeated cross-validation for the two-class classification included 94% sensitivity, 91% specificity, 93% accuracy, and 0.93 area under the ROC curve (AUC: 95% CI 0.89-0.97). Conclusion: The AI model used in this study can successfully identify CTPA exams with diagnostic interpretation limiting motion artifacts in multicenter training and test datasets. Clinical relevance: The AI model used in the study can help alert technologists about the presence of substantial motion artifacts on CTPA, where a repeat image acquisition can help salvage diagnostic information.

Evaluation of an Artificial Intelligence Model for Detection of Pneumothorax and Tension Pneumothorax in Chest Radiographs

IMPORTANCE

Early detection of pneumothorax, most often via chest radiography, can help determine need for emergent clinical intervention. The ability to accurately detect and rapidly triage pneumothorax with an artificial intelligence (AI) model could assist with earlier identification and improve care.

OBJECTIVE

To compare the accuracy of an AI model vs consensus thoracic radiologist interpretations in detecting any pneumothorax (incorporating both nontension and tension pneumothorax) and tension pneumothorax.

DESIGN, SETTING, AND PARTICIPANTS

This diagnostic study was a retrospective standalone performance assessment using a data set of 1000 chest radiographs captured between June 1, 2015, and May 31, 2021. The radiographs were obtained from patients aged at least 18 years at 4 hospitals in the Mass General Brigham hospital network in the United States. Included radiographs were selected using 2 strategies from all chest radiography performed at the hospitals, including inpatient and outpatient. The first strategy identified consecutive radiographs with pneumothorax through a manual review of radiology reports, and the second strategy identified consecutive radiographs with tension pneumothorax using natural language processing. For both strategies, negative radiographs were selected by taking the next negative radiograph acquired from the same radiography machine as each positive radiograph. The final data set was an amalgamation of these processes. Each radiograph was interpreted independently by up to 3 radiologists to establish consensus ground-truth interpretations. Each radiograph was then interpreted by the AI model for the presence of pneumothorax and tension pneumothorax. This study was conducted between July and October 2021, with the primary analysis performed between October and November 2021.

MAIN OUTCOMES AND MEASURES

The primary end points were the areas under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. The secondary end points were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax.

RESULTS

The final analysis included radiographs from 985 patients (mean [SD] age, 60.8 [19.0] years; 436 [44.3%] female patients), including 307 patients with nontension pneumothorax, 128 patients with tension pneumothorax, and 550 patients without pneumothorax. The AI model detected any pneumothorax with an AUC of 0.979 (95% CI, 0.970-0.987), sensitivity of 94.3% (95% CI, 92.0%-96.3%), and specificity of 92.0% (95% CI, 89.6%-94.2%) and tension pneumothorax with an AUC of 0.987 (95% CI, 0.980-0.992), sensitivity of 94.5% (95% CI, 90.6%-97.7%), and specificity of 95.3% (95% CI, 93.9%-96.6%).

CONCLUSIONS AND RELEVANCE

These findings suggest that the assessed AI model accurately detected pneumothorax and tension pneumothorax in this chest radiograph data set. The model's use in the clinical workflow could lead to earlier identification and improved care for patients with pneumothorax.

Use of Artificial Intelligence in Clinical Neurology

Artificial intelligence is already innovating in the provision of neurologic care. This review explores key artificial intelligence concepts; their application to neurologic diagnosis, prognosis, and treatment; and challenges that await their broader adoption. The development of new diagnostic biomarkers, individualization of prognostic information, and improved access to treatment are among the plethora of possibilities. These advances, however, reflect only the tip of the iceberg for the ways in which artificial intelligence may transform neurologic care in the future.

Development and clinical application of a deep learning model to identify acute infarct on magnetic resonance imaging

Stroke is a leading cause of death and disability. The ability to quickly identify the presence of acute infarct and quantify the volume on magnetic resonance imaging (MRI) has important treatment implications. We developed a machine learning model that used the apparent diffusion coefficient and diffusion weighted imaging series. It was trained on 6,657 MRI studies from Massachusetts General Hospital (MGH; Boston, USA). All studies were labelled positive or negative for infarct (classification annotation) with 377 having the region of interest outlined (segmentation annotation). The different annotation types facilitated training on more studies while not requiring the extensive time to manually segment every study. We initially validated the model on studies sequestered from the training set. We then tested the model on studies from three clinical scenarios: consecutive stroke team activations for 6-months at MGH, consecutive stroke team activations for 6-months at a hospital that did not provide training data (Brigham and Women’s Hospital [BWH]; Boston, USA), and an international site (Diagnósticos da América SA [DASA]; Brazil). The model results were compared to radiologist ground truth interpretations. The model performed better when trained on classification and segmentation annotations (area under the receiver operating curve [AUROC] 0.995 [95% CI 0.992–0.998] and median Dice coefficient for segmentation overlap of 0.797 [IQR 0.642–0.861]) compared to segmentation annotations alone (AUROC 0.982 [95% CI 0.972–0.990] and Dice coefficient 0.776 [IQR 0.584–0.857]). The model accurately identified infarcts for MGH stroke team activations (AUROC 0.964 [95% CI 0.943–0.982], 381 studies), BWH stroke team activations (AUROC 0.981 [95% CI 0.966–0.993], 247 studies), and at DASA (AUROC 0.998 [95% CI 0.993–1.000], 171 studies). The model accurately segmented infarcts with Pearson correlation comparing model output and ground truth volumes between 0.968 and 0.986 for the three scenarios. Acute infarct can be accurately detected and segmented on MRI in real-world clinical scenarios using a machine learning model.

Consensus disease definitions for the spectrum of neurologic immune related adverse events

2647

Background: Expanding FDA-approved indications for immune checkpoint inhibitors in patients with cancer has resulted in both therapeutic success and immune related adverse events (irAEs). Neurologic irAEs (irAE-Ns) have an incidence of 1-12% and a high fatality rate relative to other irAEs. Lack of standardized disease definitions and accurate phenotyping leads to syndrome misclassification and impedes evidence-based treatments and research progress. The objectives of this study were to develop consensus guidance for an approach to irAE-Ns including disease definitions and severity grading. Methods: A working group of 4 neurologists drafted irAE-N consensus guidance and definitions, which were reviewed by the Neuro irAE Disease Definition Panel, consisting of neurologists, oncologists, neuro-oncologists and irAE subspecialists. A modified Delphi consensus process was used, with 2 rounds of anonymous ratings by panelists and 2 virtual meetings to discuss areas of controversy. Panelists rated content for usability, appropriateness and accuracy on 9-point scales in electronic surveys and provided free text comments. The working group aggregated survey responses and incorporated them into revised definitions. Consensus was based on numeric ratings using the RAND/UCLA Appropriateness Method with prespecified definitions. Results: Twenty-seven panelists from 15 academic medical centers voted on a total of 53 rating scales (6 general guidance, 24 central and 18 peripheral nervous system disease definition components, 3 severity criteria and 2 clinical trial adjudication statements); of these, 77% (41/53) received first round consensus. After revisions, all items received second round consensus. Consensus definitions were achieved for 7 core disorders: irMeningitis, irEncephalitis/Encephalomyelitis, irDemyelinating disease, irVasculitis, irNeuropathy, irNeuromuscular junction disorders and irMyopathy. For each disorder, 6 sub-classifications are described: disease subtype, diagnostic certainty, severity, autoantibody association, exacerbation of pre-existing disease or de novo presentation and present or absent concurrent irAE. Conclusions: These disease definitions standardize irAE-N classification. They are being incorporated into a multi-institutional registry that our group has initiated to study irAEs. Given consensus on their accuracy and usability from a representative panel group, we anticipate that they can be used broadly across clinical and research settings.

Unilateral Relapsing Primary Angiitis of the CNS: An Entity Suggesting Differences in the Immune Response Between the Cerebral Hemispheres

OBJECTIVE

To determine whether studying patients with strictly unilateral relapsing primary angiitis of the CNS (UR-PACNS) can support hemispheric differences in immune response mechanisms, we reviewed characteristics of a group of such patients.

METHODS

We surveiled our institution for patients with UR-PACNS, after characterizing one such case. We defined UR-PACNS as PACNS with clinical and radiographic relapses strictly recurring in 1 brain hemisphere, with or without hemiatrophy. PACNS must have been biopsy proven. Three total cases were identified at our institution. A literature search for similar reports yielded 4 additional cases. The combined 7 cases were reviewed for demographic, clinical, imaging, and pathologic trends.

RESULTS

The median age at time of clinical onset among the 7 cases was 26 years (range 10-49 years); 5 were male (71%). All 7 patients presented with seizures. The mean follow-up duration was 7.5 years (4-14.1 years). The annualized relapse rate ranged between 0.2 and 1. UR-PACNS involved the left cerebral hemisphere in 5 of the 7 patients. There was no consistent relationship between the patient's dominant hand and the diseased side. When performed (5 cases), conventional angiogram was nondiagnostic. CSF examination showed nucleated cells and protein levels in normal range in 3 cases and ranged from 6 to 11 cells/μL and 49 to 110 mg/dL in 4 cases, respectively. All cases were diagnosed with lesional biopsy, showing lymphocytic type of vasculitis of the small- and medium-sized vessels. Patients treated with steroids alone showed progression. Induction therapy with cyclophosphamide or rituximab followed by a steroid sparing agent resulted in the most consistent disease remission.

CONCLUSIONS

Combining our 3 cases with others reported in the literature allows better clinical understanding about this rare and extremely puzzling disease entity. We hypothesize that a functional difference in immune responses, caused by such discrepancies as basal levels of cytokines, asymmetric distribution of microglia, and differences in modulation of the systemic immune functions, rather than a structural antigenic difference, between the right and left brain may explain this phenomenon, but this is speculative.

Neuromyelitis optica after splenectomy: A secondary autoimmune phenomenon

We describe the case of a 53-year-old woman who undergoes total splenectomy and later presents with aquaporin-4 antibody positive neuromyelitis optica (NMO). The occurrence of NMO after acquired immunosuppression raises the possibility of NMO as a form of secondary autoimmunity.

Neurology Training Worldwide

Neurology training is essential for providing neurologic care globally. Large disparities in availability of neurology training exist between higher- and lower-income countries. This review explores the worldwide distribution of neurology training programs and trainees, the characteristics of training programs in different parts of the world, and initiatives aimed at increasing access to neurology training in under-resourced regions.

Cuprizone demyelination induces a unique inflammatory response in the subventricular zone

Background

Cuprizone leads to demyelination of the corpus callosum (CC) and activates progenitor cells in the adjacent subventricular zone (SVZ), a stem cell niche which contributes to remyelination. The healthy SVZ contains semi-activated microglia and constitutively expresses the pro-inflammatory molecule galectin-3 (Gal-3) suggesting the niche uniquely regulates inflammation.

Methods

We studied the inflammatory response to cuprizone in the SVZ and CC in Gal-3 knockout mice using immunohistochemistry and with the in vitro neurosphere assay.

Results

Cuprizone caused loss of myelin basic protein (MBP) immunofluorescence in the CC suggesting demyelination. Cuprizone increased the density of CD45+/Iba1+ microglial cells and also increased Gal-3 expression in the CC. Surprisingly, the number of Gal-3+ and CD45+ cells decreased in the SVZ after cuprizone, suggesting inflammation was selectively reduced therein. Inflammation can regulate SVZ proliferation and indeed the number of phosphohistone H3+ (PHi3+) cells decreased in the SVZ but increased in the CC in both genotypes after cuprizone treatment. BrdU+ SVZ cell numbers also decreased in the SVZ after cuprizone, and this effect was significantly greater at 3 weeks in Gal-3^−/− mice compared to WT, suggesting Gal-3 normally limits SVZ cell emigration following cuprizone treatment.

Conclusions

This study reveals a uniquely regulated inflammatory response in the SVZ and shows that Gal-3 participates in remyelination in the cuprizone model. This contrasts with more severe models of demyelination which induce SVZ inflammation and suggests the extent of demyelination affects the SVZ neurogenic response.

Electronic supplementary material

The online version of this article (doi:10.1186/s12974-016-0651-2) contains supplementary material, which is available to authorized users.

The surface of the empirical horopter

The distribution of empirical corresponding points in the two retinas has been well studied along the horizontal and the vertical meridians, but not in other parts of the visual field. Using an apparent-motion paradigm, we measured the positions of those points across the central portion of the visual field. We found that the Hering-Hillebrand deviation (a deviation from the Vieth-Müller circle) and the Helmholtz shear of horizontal disparity (backward slant of the vertical horopter) exist throughout the visual field. We also found no evidence for non-zero vertical disparities in empirical corresponding points. We used the data to find the combination of points in space and binocular eye position that minimizes the disparity between stimulated points on the retinas and the empirical corresponding points. The optimum surface is a top-back slanted surface at medium to far distance depending on the observer. The line in the middle of the surface extending away from the observer comes very close to lying in the plane of the ground as the observer fixates various positions in the ground, a speculation Helmholtz made that has since been misunderstood.

The relation between color discrimination and color constancy: when is optimal adaptation task dependent?

Color vision supports two distinct visual functions: discrimination and constancy. Discrimination requires that the visual response to distinct objects within a scene be different. Constancy requires that the visual response to any object be the same across scenes. Across changes in scene, adaptation can improve discrimination by optimizing the use of the available response range. Similarly, adaptation can improve constancy by stabilizing the visual response to any fixed object across changes in illumination. Can common mechanisms of adaptation achieve these two goals simultaneously? We develop a theoretical framework for answering this question and present several example calculations. In the examples studied, the answer is largely yes when the change of scene consists of a change in illumination and considerably less so when the change of scene consists of a change in the statistical ensemble of surface reflectances in the environment.

Distinct mechanisms mediate visual detection and identification

A core organizing principle for studies of the brain is that distinct neural pathways mediate distinct behavioral tasks [1, 2]. When two related tasks are mediated by a common pathway, studies of one are likely to generalize to the other. Here, we test whether performance on two laboratory tasks that model object detection and identification are mediated by common mechanisms of visual adaptation. Although both tasks rely on the luminance pattern in images, their demands on visual processing are quite different. Object detection requires discriminating image luminance differences associated with the light reflected from adjacent objects. To encode these differences reliably, neurons adapt their limited dynamic range to prevailing viewing conditions [3-6]. Object identification, on the other hand, demands a fixed response to light reflected from an object independent of illumination [7]. We compared performance in discrimination and identification tasks for simulated surfaces. In striking contrast to studies with less structured contexts, we found clear evidence that distinct processes mediate judgments in the two tasks. These results challenge models that account for perceived lightness entirely through the action of image-encoding mechanisms.

Do common mechanisms of adaptation mediate color discrimination and appearance? Contrast adaptation

Are effects of background contrast on color appearance and sensitivity controlled by the same mechanism of adaptation? We examined the effects of background color contrast on color appearance and on color-difference sensitivity under well-matched conditions. We linked the data using Fechner's hypothesis that the rate of apparent stimulus change is proportional to sensitivity and examined a family of parametric models of adaptation. Our results show that both appearance and discrimination are consistent with the same mechanism of adaptation.

Do common mechanisms of adaptation mediate color discrimination and appearance? Uniform backgrounds

Color vision is useful for detecting surface boundaries and identifying objects. Are the signals used to perform these two functions processed by common mechanisms, or has the visual system optimized its processing separately for each task? We measured the effect of mean chromaticity and luminance on color discriminability and on color appearance under well-matched stimulus conditions. In the discrimination experiments, a pedestal spot was presented in one interval and a pedestal + test in a second. Observers indicated which interval contained the test. In the appearance experiments, observers matched the appearance of test spots across a change in background. We analyzed the data using a variant of Fechner's proposal, that the rate of apparent stimulus change is proportional to visual sensitivity. We found that saturating visual response functions together with a model of adaptation that included multiplicative gain control and a subtractive term accounted for data from both tasks. This result suggests that effects of the contexts we studied on color appearance and discriminability are controlled by the same underlying mechanism.

Slant from texture and disparity cues: optimal cue combination

How does the visual system combine information from different depth cues to estimate three-dimensional scene parameters? We tested a maximum-likelihood estimation (MLE) model of cue combination for perspective (texture) and binocular disparity cues to surface slant. By factoring the reliability of each cue into the combination process, MLE provides more reliable estimates of slant than would be available from either cue alone. We measured the reliability of each cue in isolation across a range of slants and distances using a slant-discrimination task. The reliability of the texture cue increases as |slant| increases and does not change with distance. The reliability of the disparity cue decreases as distance increases and varies with slant in a way that also depends on viewing distance. The trends in the single-cue data can be understood in terms of the information available in the retinal images and issues related to solving the binocular correspondence problem. To test the MLE model, we measured perceived slant of two-cue stimuli when disparity and texture were in conflict and the reliability of slant estimation when both cues were available. Results from the two-cue study indicate, consistent with the MLE model, that observers weight each cue according to its relative reliability: Disparity weight decreased as distance and |slant| increased. We also observed the expected improvement in slant estimation when both cues were available. With few discrepancies, our data indicate that observers combine cues in a statistically optimal fashion and thereby reduce the variance of slant estimates below that which could be achieved from either cue alone. These results are consistent with other studies that quantitatively examined the MLE model of cue combination. Thus, there is a growing empirical consensus that MLE provides a good quantitative account of cue combination and that sensory information is used in a manner that maximizes the precision of perceptual estimates.

Relative image size, not eye position, determines eye dominance switches

A recent paper examined eye dominance with the eyes in forward and eccentric gaze [Vision Res. 41 (2001) 1743]. When observers were looking to the left, the left eye tended to dominate and when they were looking to the right, the right eye tended to dominate. The authors attributed the switch in eye dominance to extra-retinal signals associated with horizontal eye position. However, when one looks at a near object on the left, the image in the left eye is larger than the one in the right eye, and when one looks to the right, the opposite occurs. Thus, relative image size could also trigger switches in eye dominance. We used a cue-conflict paradigm to determine whether eye position or relative image size is the determinant of eye-dominance switches with changes in gaze angle. When eye position and relative image size were varied independently, there was no consistent effect of eye position. Relative image size appears to be the sole determinant of the switch.

Combining sensory information: mandatory fusion within, but not between, senses

Humans use multiple sources of sensory information to estimate environmental properties. For example, the eyes and hands both provide relevant information about an object's shape. The eyes estimate shape using binocular disparity, perspective projection, etc. The hands supply haptic shape information by means of tactile and proprioceptive cues. Combining information across cues can improve estimation of object properties but may come at a cost: loss of single-cue information. We report that single-cue information is indeed lost when cues from within the same sensory modality (disparity and texture gradients in vision) are combined, but not when different modalities (vision and haptics) are combined.

Are corresponding points fixed?

Several investigators have claimed that the retinal coordinates of corresponding points shift with vergence eye movements. Two kinds of shifts have been reported. First, global shifts that increase with retinal eccentricity; such shifts would cause a flattening of the horopter at all viewing distances and would facilitate fusion of flat surfaces. Second, local shifts that are centered on the fovea; such shifts would cause a dimple in the horopter near fixation and would facilitate fusion of points fixated at extreme viewing distances. Nearly all of the empirical evidence supporting shifts of corresponding points comes from horopter measurements and from comparisons of subjective and objective fixation disparity. In both cases, the experimenter must infer the retinal coordinates of corresponding points from external measurements. We describe four factors that could affect this inference: (1) changes in the projection from object to image points that accompany eye rotation and accommodation, (2) fixation errors during the experimental measurements, (3) non-uniform retinal stretching, and (4) changes in the perceived direction of a monocular point when presented adjacent to a binocular point. We conducted two experiments that eliminated or compensated for these potential errors. In the first experiment, observers aligned dichoptic test lines using an apparatus and procedure that eliminated all but the third error. In the second experiment, observers judged the alignment of dichoptic afterimages, and this technique eliminates all the errors. The results from both experiments show that the retinal coordinates of corresponding points do not change with vergence eye movements. We conclude that corresponding points are in fixed retinal positions for observers with normal retinal correspondence.