DeepDicomSort: An Automatic Sorting Algorithm for Brain Magnetic Resonance Imaging Data

Explainable AI in Diagnostic Radiology for Neurological Disorders: A Systematic Review, and What Doctors Think About It

Background: Artificial intelligence (AI) has recently made unprecedented contributions in every walk of life, but it has not been able to work its way into diagnostic medicine and standard clinical practice yet. Although data scientists, researchers, and medical experts have been working in the direction of designing and developing computer aided diagnosis (CAD) tools to serve as assistants to doctors, their large-scale adoption and integration into the healthcare system still seems far-fetched. Diagnostic radiology is no exception. Imagining techniques like magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET) scans have been widely and very effectively employed by radiologists and neurologists for the differential diagnoses of neurological disorders for decades, yet no AI-powered systems to analyze such scans have been incorporated into the standard operating procedures of healthcare systems. Why? It is absolutely understandable that in diagnostic medicine, precious human lives are on the line, and hence there is no room even for the tiniest of mistakes. Nevertheless, with the advent of explainable artificial intelligence (XAI), the old-school black boxes of deep learning (DL) systems have been unraveled. Would XAI be the turning point for medical experts to finally embrace AI in diagnostic radiology? This review is a humble endeavor to find the answers to these questions. Methods: In this review, we present the journey and contributions of AI in developing systems to recognize, preprocess, and analyze brain MRI scans for differential diagnoses of various neurological disorders, with special emphasis on CAD systems embedded with explainability. A comprehensive review of the literature from 2017 to 2024 was conducted using host databases. We also present medical domain experts' opinions and summarize the challenges up ahead that need to be addressed in order to fully exploit the tremendous potential of XAI in its application to medical diagnostics and serve humanity. Results: Forty-seven studies were summarized and tabulated with information about the XAI technology and datasets employed, along with performance accuracies. The strengths and weaknesses of the studies have also been discussed. In addition, the opinions of seven medical experts from around the world have been presented to guide engineers and data scientists in developing such CAD tools. Conclusions: Current CAD research was observed to be focused on the enhancement of the performance accuracies of the DL regimens, with less attention being paid to the authenticity and usefulness of explanations. A shortage of ground truth data for explainability was also observed. Visual explanation methods were found to dominate; however, they might not be enough, and more thorough and human professor-like explanations would be required to build the trust of healthcare professionals. Special attention to these factors along with the legal, ethical, safety, and security issues can bridge the current gap between XAI and routine clinical practice.

An Automated Tool to Classify and Transform Unstructured MRI Data into BIDS Datasets

The increasing use of neuroimaging in clinical research has driven the creation of many large imaging datasets. However, these datasets often rely on inconsistent naming conventions in image file headers to describe acquisition, and time-consuming manual curation is necessary. Therefore, we sought to automate the process of classifying and organizing magnetic resonance imaging (MRI) data according to acquisition types common to the clinical routine, as well as automate the transformation of raw, unstructured images into Brain Imaging Data Structure (BIDS) datasets. To do this, we trained an XGBoost model to classify MRI acquisition types using relatively few acquisition parameters that are automatically stored by the MRI scanner in image file metadata, which are then mapped to the naming conventions prescribed by BIDS to transform the input images to the BIDS structure. The model recognizes MRI types with 99.475% accuracy, as well as a micro/macro-averaged precision of 0.9995/0.994, a micro/macro-averaged recall of 0.9995/0.989, and a micro/macro-averaged F1 of 0.9995/0.991. Our approach accurately and quickly classifies MRI types and transforms unstructured data into standardized structures with little-to-no user intervention, reducing the barrier of entry for clinical scientists and increasing the accessibility of existing neuroimaging data.

Brain MRI sequence and view plane identification using deep learning

Brain magnetic resonance imaging (MRI) scans are available in a wide variety of sequences, view planes, and magnet strengths. A necessary preprocessing step for any automated diagnosis is to identify the MRI sequence, view plane, and magnet strength of the acquired image. Automatic identification of the MRI sequence can be useful in labeling massive online datasets used by data scientists in the design and development of computer aided diagnosis (CAD) tools. This paper presents a deep learning (DL) approach for brain MRI sequence and view plane identification using scans of different data types as input. A 12-class classification system is presented for commonly used MRI scans, including T1, T2-weighted, proton density (PD), fluid attenuated inversion recovery (FLAIR) sequences in axial, coronal and sagittal view planes. Multiple online publicly available datasets have been used to train the system, with multiple infrastructures. MobileNet-v2 offers an adequate performance accuracy of 99.76% with unprocessed MRI scans and a comparable accuracy with skull-stripped scans and has been deployed in a tool for public use. The tool has been tested on unseen data from online and hospital sources with a satisfactory performance accuracy of 99.84 and 86.49%, respectively.

Deep Learning-based Identification of Brain MRI Sequences Using a Model Trained on Large Multicentric Study Cohorts

Purpose To develop a fully automated device- and sequence-independent convolutional neural network (CNN) for reliable and high-throughput labeling of heterogeneous, unstructured MRI data. Materials and Methods Retrospective, multicentric brain MRI data (2179 patients with glioblastoma, 8544 examinations, 63 327 sequences) from 249 hospitals and 29 scanner types were used to develop a network based on ResNet-18 architecture to differentiate nine MRI sequence types, including T1-weighted, postcontrast T1-weighted, T2-weighted, fluid-attenuated inversion recovery, susceptibility-weighted, apparent diffusion coefficient, diffusion-weighted (low and high b value), and gradient-recalled echo T2*-weighted and dynamic susceptibility contrast-related images. The two-dimensional-midsection images from each sequence were allocated to training or validation (approximately 80%) and testing (approximately 20%) using a stratified split to ensure balanced groups across institutions, patients, and MRI sequence types. The prediction accuracy was quantified for each sequence type, and subgroup comparison of model performance was performed using χ2 tests. Results On the test set, the overall accuracy of the CNN (ResNet-18) ensemble model among all sequence types was 97.9% (95% CI: 97.6, 98.1), ranging from 84.2% for susceptibility-weighted images (95% CI: 81.8, 86.6) to 99.8% for T2-weighted images (95% CI: 99.7, 99.9). The ResNet-18 model achieved significantly better accuracy compared with ResNet-50 despite its simpler architecture (97.9% vs 97.1%; P ≤ .001). The accuracy of the ResNet-18 model was not affected by the presence versus absence of tumor on the two-dimensional-midsection images for any sequence type (P > .05). Conclusion The developed CNN (www.github.com/neuroAI-HD/HD-SEQ-ID) reliably differentiates nine types of MRI sequences within multicenter and large-scale population neuroimaging data and may enhance the speed, accuracy, and efficiency of clinical and research neuroradiologic workflows. Keywords: MR-Imaging, Neural Networks, CNS, Brain/Brain Stem, Computer Applications-General (Informatics), Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms Supplemental material is available for this article. © RSNA, 2023.

Sequence-Type Classification of Brain MRI for Acute Stroke Using a Self-Supervised Machine Learning Algorithm

We propose a self-supervised machine learning (ML) algorithm for sequence-type classification of brain MRI using a supervisory signal from DICOM metadata (i.e., a rule-based virtual label). A total of 1787 brain MRI datasets were constructed, including 1531 from hospitals and 256 from multi-center trial datasets. The ground truth (GT) was generated by two experienced image analysts and checked by a radiologist. An ML framework called ImageSort-net was developed using various features related to MRI acquisition parameters and used for training virtual labels and ML algorithms derived from rule-based labeling systems that act as labels for supervised learning. For the performance evaluation of ImageSort-net (MLvirtual), we compare and analyze the performances of models trained with human expert labels (MLhumans), using as a test set blank data that the rule-based labeling system failed to infer from each dataset. The performance of ImageSort-net (MLvirtual) was comparable to that of MLhuman (98.5% and 99%, respectively) in terms of overall accuracy when trained with hospital datasets. When trained with a relatively small multi-center trial dataset, the overall accuracy was relatively lower than that of MLhuman (95.6% and 99.4%, respectively). After integrating the two datasets and re-training them, MLvirtual showed higher accuracy than MLvirtual trained only on multi-center datasets (95.6% and 99.7%, respectively). Additionally, the multi-center dataset inference performances after the re-training of MLvirtual and MLhumans were identical (99.7%). Training of ML algorithms based on rule-based virtual labels achieved high accuracy for sequence-type classification of brain MRI and enabled us to build a sustainable self-learning system.

Association of pre-radiotherapy tumour burden and overall survival in newly diagnosed glioblastoma adjusted for MGMT promoter methylation status

PURPOSE

We retrospectively evaluated the association between postoperative pre-radiotherapy tumour burden and overall survival (OS) adjusted for the prognostic value of O6-methylguanine DNA methyltransferase (MGMT) promoter methylation in patients with newly diagnosed glioblastoma treated with radio-/chemotherapy with temozolomide.

MATERIALS AND METHODS

Patients were included from the CENTRIC (EORTC 26071-22072) and CORE trials if postoperative magnetic resonance imaging scans were available within a timeframe of up to 4weeks before radiotherapy, including both pre- and post-contrast T1w images and at least one T2w sequence (T2w or T2w-FLAIR). Postoperative (residual) pre-radiotherapy contrast-enhanced tumour (CET) volumes and non-enhanced T2w abnormalities (NT2A) tissue volumes were obtained by three-dimensional segmentation. Cox proportional hazard models and Kaplan Meier estimates were used to assess the association of pre-radiotherapy CET/NT2A volume with OS adjusted for known prognostic factors (age, performance status, MGMT status).

RESULTS

408 tumour (of which 270 MGMT methylated) segmentations were included. Median OS in patients with MGMT methylated tumours was 117 weeks versus 61weeks in MGMT unmethylated tumours (p < 0.001). When stratified for MGMT methylation status, higher CET volume (HR 1.020; 95% confidence interval CI [1.013-1.027]; p < 0.001) and older age (HR 1.664; 95% CI [1.214-2.281]; p = 0.002) were significantly associated with shorter OS while NT2A volume and performance status were not.

CONCLUSION

Pre-radiotherapy CET volume was strongly associated with OS in patients receiving radio-/chemotherapy for newly diagnosed glioblastoma stratified by MGMT promoter methylation status.

Integrative Imaging Informatics for Cancer Research: Workflow Automation for Neuro-Oncology (I3CR-WANO)

PURPOSE

Efforts to use growing volumes of clinical imaging data to generate tumor evaluations continue to require significant manual data wrangling, owing to data heterogeneity. Here, we propose an artificial intelligence-based solution for the aggregation and processing of multisequence neuro-oncology MRI data to extract quantitative tumor measurements.

MATERIALS AND METHODS

Our end-to-end framework (1) classifies MRI sequences using an ensemble classifier, (2) preprocesses the data in a reproducible manner, (3) delineates tumor tissue subtypes using convolutional neural networks, and (4) extracts diverse radiomic features. Moreover, it is robust to missing sequences and adopts an expert-in-the-loop approach in which the segmentation results may be manually refined by radiologists. After the implementation of the framework in Docker containers, it was applied to two retrospective glioma data sets collected from the Washington University School of Medicine (WUSM; n = 384) and The University of Texas MD Anderson Cancer Center (MDA; n = 30), comprising preoperative MRI scans from patients with pathologically confirmed gliomas.

RESULTS

The scan-type classifier yielded an accuracy of >99%, correctly identifying sequences from 380 of 384 and 30 of 30 sessions from the WUSM and MDA data sets, respectively. Segmentation performance was quantified using the Dice Similarity Coefficient between the predicted and expert-refined tumor masks. The mean Dice scores were 0.882 (±0.244) and 0.977 (±0.04) for whole-tumor segmentation for WUSM and MDA, respectively.

CONCLUSION

This streamlined framework automatically curated, processed, and segmented raw MRI data of patients with varying grades of gliomas, enabling the curation of large-scale neuro-oncology data sets and demonstrating high potential for integration as an assistive tool in clinical practice.

MR-Class: A Python Tool for Brain MR Image Classification Utilizing One-vs-All DCNNs to Deal with the Open-Set Recognition Problem

Background: MR image classification in datasets collected from multiple sources is complicated by inconsistent and missing DICOM metadata. Therefore, we aimed to establish a method for the efficient automatic classification of MR brain sequences. Methods: Deep convolutional neural networks (DCNN) were trained as one-vs-all classifiers to differentiate between six classes: T1 weighted (w), contrast-enhanced T1w, T2w, T2w-FLAIR, ADC, and SWI. Each classifier yields a probability, allowing threshold-based and relative probability assignment while excluding images with low probability (label: unknown, open-set recognition problem). Data from three high-grade glioma (HGG) cohorts was assessed; C1 (320 patients, 20,101 MRI images) was used for training, while C2 (197, 11,333) and C3 (256, 3522) were for testing. Two raters manually checked images through an interactive labeling tool. Finally, MR-Class’ added value was evaluated via radiomics model performance for progression-free survival (PFS) prediction in C2, utilizing the concordance index (C-I). Results: Approximately 10% of annotation errors were observed in each cohort between the DICOM series descriptions and the derived labels. MR-Class accuracy was 96.7% [95% Cl: 95.8, 97.3] for C2 and 94.4% [93.6, 96.1] for C3. A total of 620 images were misclassified; manual assessment of those frequently showed motion artifacts or alterations of anatomy by large tumors. Implementation of MR-Class increased the PFS model C-I by 14.6% on average, compared to a model trained without MR-Class. Conclusions: We provide a DCNN-based method for the sequence classification of brain MR images and demonstrate its usability in two independent HGG datasets.

Deep learning-based end-to-end scan-type classification, pre-processing, and segmentation of clinical neuro-oncology studies

Modern neuro-oncology workflows are driven by large collections of high-dimensional MRI data obtained using varying acquisition protocols. The concomitant heterogeneity of this data makes extensive manual curation and pre-processing imperative prior to algorithmic use. The limited efforts invested towards automating this curation and processing are fragmented, do not encompass the entire workflow, or still require significant manual intervention. In this work, we propose an artificial intelligence-driven solution for transforming multi-modal raw neuro-oncology MRI Digital Imaging and Communications in Medicine (DICOM) data into quantitative tumor measurements. Our end-to-end framework classifies MRI scans into different structural sequence types, preprocesses the data, and uses convolutional neural networks to segment tumor tissue subtypes. Moreover, it adopts an expert-in-the-loop approach, where segmentation results may be manually refined by radiologists. This framework was implemented as Docker Containers (for command line usage and within the eXtensible Neuroimaging Archive Toolkit [XNAT]) and validated on a retrospective glioma dataset (n = 155) collected from the Washington University School of Medicine, comprising preoperative MRI scans from patients with histopathologically confirmed gliomas. Segmentation results were refined by a neuroradiologist, and performance was quantified using Dice Similarity Coefficient to compare predicted and expert-refined tumor masks. The scan-type classifier yielded a 99.71% accuracy across all sequence types. The segmentation model achieved mean Dice scores of 0.894 (± 0.225) for whole tumor segmentation. The proposed framework can automate tumor segmentation and characterization - thus streamlining workflows in a clinical setting as well as expediting standardized curation of large-scale neuro-oncology datasets in a research setting.

Deep multi-task learning and random forest for series classification by pulse sequence type and orientation

PURPOSE

Increasingly complex MRI studies and variable series naming conventions reveal limitations of rule-based image routing, especially in health systems with multiple scanners and sites. Accurate methods to identify series based on image content would aid post-processing and PACS viewing. Recent deep/machine learning efforts classify 5-8 basic brain MR sequences. We present an ensemble model combining a convolutional neural network and a random forest classifier to differentiate 25 brain sequences and image orientation.

METHODS

Series were grouped by descriptions into 25 sequences and 4 orientations. Dataset A, obtained from our institution, was divided into training (16,828 studies; 48,512 series; 112,028 images), validation (4746 studies; 16,612 series; 26,222 images) and test sets (6348 studies; 58,705 series; 3,314,018 images). Dataset B, obtained from a separate hospital, was used for out-of-domain external validation (1252 studies; 2150 series; 234,944 images). We developed an ensemble model combining a 2D convolutional neural network with a custom multi-task learning architecture and random forest classifier trained on DICOM metadata to classify sequence and orientation by series.

RESULTS

The neural network, random forest, and ensemble achieved 95%, 97%, and 98% overall sequence accuracy on dataset A, and 98%, 99%, and 99% accuracy on dataset B, respectively. All models achieved > 99% orientation accuracy on both datasets.

CONCLUSION

The ensemble model for series identification accommodates the complexity of brain MRI studies in state-of-the-art clinical practice. Expanding on previous work demonstrating proof-of-concept, our approach is more comprehensive with greater sequence diversity and orientation classification.

Challenges and opportunities for advanced neuroimaging of glioblastoma

Glioblastoma is the most aggressive of glial tumours in adults. On conventional magnetic resonance (MR) imaging, these tumours are observed as irregular enhancing lesions with areas of infiltrating tumour and cortical expansion. More advanced imaging techniques including diffusion-weighted MRI, perfusion-weighted MRI, MR spectroscopy and positron emission tomography (PET) imaging have found widespread application to diagnostic challenges in the setting of first diagnosis, treatment planning and follow-up. This review aims to educate readers with regard to the strengths and weaknesses of the clinical application of these imaging techniques. For example, this review shows that the (semi)quantitative analysis of the mentioned advanced imaging tools was found useful for assessing tumour aggressiveness and tumour extent, and aids in the differentiation of tumour progression from treatment-related effects. Although these techniques may aid in the diagnostic work-up and (post-)treatment phase of glioblastoma, so far no unequivocal imaging strategy is available. Furthermore, the use and further development of artificial intelligence (AI)-based tools could greatly enhance neuroradiological practice by automating labour-intensive tasks such as tumour measurements, and by providing additional diagnostic information such as prediction of tumour genotype. Nevertheless, due to the fact that advanced imaging and AI-diagnostics is not part of response assessment criteria, there is no harmonised guidance on their use, while at the same time the lack of standardisation severely hampers the definition of uniform guidelines.

CardiSort: a convolutional neural network for cross vendor automated sorting of cardiac MR images

OBJECTIVES

To develop an image-based automatic deep learning method to classify cardiac MR images by sequence type and imaging plane for improved clinical post-processing efficiency.

METHODS

Multivendor cardiac MRI studies were retrospectively collected from 4 centres and 3 vendors. A two-head convolutional neural network ('CardiSort') was trained to classify 35 sequences by imaging sequence (n = 17) and plane (n = 10). Single vendor training (SVT) on single-centre images (n = 234 patients) and multivendor training (MVT) with multicentre images (n = 434 patients, 3 centres) were performed. Model accuracy and F1 scores on a hold-out test set were calculated, with ground truth labels by an expert radiologist. External validation of MVT (MVTexternal) was performed on data from 3 previously unseen magnet systems from 2 vendors (n = 80 patients).

RESULTS

Model sequence/plane/overall accuracy and F1-scores were 85.2%/93.2%/81.8% and 0.82 for SVT and 96.1%/97.9%/94.3% and 0.94 MVT on the hold-out test set. MVTexternal yielded sequence/plane/combined accuracy and F1-scores of 92.7%/93.0%/86.6% and 0.86. There was high accuracy for common sequences and conventional cardiac planes. Poor accuracy was observed for underrepresented classes and sequences where there was greater variability in acquisition parameters across centres, such as perfusion imaging.

CONCLUSIONS

A deep learning network was developed on multivendor data to classify MRI studies into component sequences and planes, with external validation. With refinement, it has potential to improve workflow by enabling automated sequence selection, an important first step in completely automated post-processing pipelines.

KEY POINTS

• Deep learning can be applied for consistent and efficient classification of cardiac MR image types. • A multicentre, multivendor study using a deep learning algorithm (CardiSort) showed high classification accuracy on a hold-out test set with good generalisation to images from previously unseen magnet systems. • CardiSort has potential to improve clinical workflows, as a vital first step in developing fully automated post-processing pipelines.

The Essential Role of Open Data and Software for the Future of Ultrasound-Based Neuronavigation

With the recent developments in machine learning and modern graphics processing units (GPUs), there is a marked shift in the way intra-operative ultrasound (iUS) images can be processed and presented during surgery. Real-time processing of images to highlight important anatomical structures combined with in-situ display, has the potential to greatly facilitate the acquisition and interpretation of iUS images when guiding an operation. In order to take full advantage of the recent advances in machine learning, large amounts of high-quality annotated training data are necessary to develop and validate the algorithms. To ensure efficient collection of a sufficient number of patient images and external validity of the models, training data should be collected at several centers by different neurosurgeons, and stored in a standard format directly compatible with the most commonly used machine learning toolkits and libraries. In this paper, we argue that such effort to collect and organize large-scale multi-center datasets should be based on common open source software and databases. We first describe the development of existing open-source ultrasound based neuronavigation systems and how these systems have contributed to enhanced neurosurgical guidance over the last 15 years. We review the impact of the large number of projects worldwide that have benefited from the publicly available datasets “Brain Images of Tumors for Evaluation” (BITE) and “Retrospective evaluation of Cerebral Tumors” (RESECT) that include MR and US data from brain tumor cases. We also describe the need for continuous data collection and how this effort can be organized through the use of a well-adapted and user-friendly open-source software platform that integrates both continually improved guidance and automated data collection functionalities.