Toward Foundational Deep Learning Models for Medical Imaging in the New Era of Transformer Networks

Surgical planning in virtual reality: a systematic review

Purpose

Virtual reality (VR) technology has emerged as a promising tool for physicians, offering the ability to assess anatomical data in 3D with visuospatial interaction qualities. The last decade has witnessed a remarkable increase in the number of studies focusing on the application of VR to assess patient-specific image data. This systematic review aims to provide an up-to-date overview of the latest research on VR in the field of surgical planning.

Approach

A comprehensive literature search was conducted based on the preferred reporting items for systematic reviews and meta-analyses covering the period from April 1, 2021 to May 10, 2023. It includes research articles reporting on preoperative surgical planning using patient-specific medical images in virtual reality using head-mounted displays. The review summarizes the current state of research in this field, identifying key findings, technologies, study designs, methods, and potential directions for future research.

Results

The selected studies show a positive impact on surgical decision-making and anatomy understanding compared to other visualization modalities. A substantial number of studies are reporting anecdotal evidence and case-specific outcomes. Notably, surgical planning using VR led to more frequent changes in surgical plans compared to planning with other visualization methods when surgeons reassessed their initial plans. VR demonstrated benefits in reducing planning time and improving spatial localization of pathologies.

Conclusions

Results show that the application of VR for surgical planning is still in an experimental stage but is gradually advancing toward clinical use. The diverse study designs, methodologies, and varying reporting hinder a comprehensive analysis. Some findings lack statistical evidence and rely on subjective assumptions. To strengthen evaluation, future research should focus on refining study designs, improving technical reporting, defining visual and technical proficiency requirements, and enhancing VR software usability and design. Addressing these areas could pave the way for an effective implementation of VR in clinical settings.

Empowering brain cancer diagnosis: harnessing artificial intelligence for advanced imaging insights

Artificial intelligence (AI) is increasingly being used in the medical field, specifically for brain cancer imaging. In this review, we explore how AI-powered medical imaging can impact the diagnosis, prognosis, and treatment of brain cancer. We discuss various AI techniques, including deep learning and causality learning, and their relevance. Additionally, we examine current applications that provide practical solutions for detecting, classifying, segmenting, and registering brain tumors. Although challenges such as data quality, availability, interpretability, transparency, and ethics persist, we emphasise the enormous potential of intelligent applications in standardising procedures and enhancing personalised treatment, leading to improved patient outcomes. Innovative AI solutions have the power to revolutionise neuro-oncology by enhancing the quality of routine clinical practice.

Toward the novel AI tasks in infection biology

Machine learning and artificial intelligence (AI) are becoming more common in infection biology laboratories around the world. Yet, as they gain traction in research, novel frontiers arise. Novel artificial intelligence algorithms are capable of addressing advanced tasks like image generation and question answering. However, similar algorithms can prove useful in addressing advanced questions in infection biology like prediction of host-pathogen interactions or inferring virus protein conformations. Addressing such tasks requires large annotated data sets, which are often scarce in biomedical research. In this review, I bring together several successful examples where such tasks were addressed. I underline the importance of formulating novel AI tasks in infection biology accompanied by freely available benchmark data sets to address these tasks. Furthermore, I discuss the current state of the field and potential future trends. I argue that one such trend involves AI tools becoming more versatile.

Optimizing Deep Learning for Cardiac MRI Segmentation: The Impact of Automated Slice Range Classification

RATIONALE AND OBJECTIVES

Cardiac magnetic resonance imaging is crucial for diagnosing cardiovascular diseases, but lengthy postprocessing and manual segmentation can lead to observer bias. Deep learning (DL) has been proposed for automated cardiac segmentation; however, its effectiveness is limited by the slice range selection from base to apex.

MATERIALS AND METHODS

In this study, we integrated an automated slice range classification step to identify basal to apical short-axis slices before DL-based segmentation. We employed publicly available Multi-Disease, Multi-View & Multi-Center Right Ventricular Segmentation in Cardiac MRI data set with short-axis cine data from 160 training, 40 validation, and 160 testing cases. Three classification and seven segmentation DL models were studied. The top-performing segmentation model was assessed with and without the classification model. Model validation to compare automated and manual segmentation was performed using Dice score and Hausdorff distance and clinical indices (correlation score and Bland-Altman plots).

RESULTS

The combined classification (CBAM-integrated 2D-CNN) and segmentation model (2D-UNet with dilated convolution block) demonstrated superior performance, achieving Dice scores of 0.952 for left ventricle (LV), 0.933 for right ventricle (RV), and 0.875 for myocardium, compared to the stand-alone segmentation model (0.949 for LV, 0.925 for RV, and 0.867 for myocardium). Combined classification and segmentation model showed high correlation (0.92-0.99) with manual segmentation for biventricular volumes, ejection fraction, and myocardial mass. The mean absolute difference (2.8-8.3 mL) for clinical parameters between automated and manual segmentation was within the interobserver variability range, indicating comparable performance to manual annotation.

CONCLUSION

Integrating an initial automated slice range classification step into the segmentation process improves the performance of DL-based cardiac chamber segmentation.

Strategies for Implementing Machine Learning Algorithms in the Clinical Practice of Radiology

Despite recent advancements in machine learning (ML) applications in health care, there have been few benefits and improvements to clinical medicine in the hospital setting. To facilitate clinical adaptation of methods in ML, this review proposes a standardized framework for the step-by-step implementation of artificial intelligence into the clinical practice of radiology that focuses on three key components: problem identification, stakeholder alignment, and pipeline integration. A review of the recent literature and empirical evidence in radiologic imaging applications justifies this approach and offers a discussion on structuring implementation efforts to help other hospital practices leverage ML to improve patient care. Clinical trial registration no. 04242667 © RSNA, 2024 Supplemental material is available for this article.

Deep Transfer Learning for Ethnically Distinct Populations: Prediction of Refractive Error Using Optical Coherence Tomography

INTRODUCTION

The mismatch between training and testing data distribution causes significant degradation in the deep learning model performance in multi-ethnic scenarios. To reduce the performance differences between ethnic groups and image domains, we built a deep transfer learning model with adaptation training to predict uncorrected refractive errors using posterior segment optical coherence tomography (OCT) images of the macula and optic nerve.

METHODS

Observational, cross-sectional, multicenter study design. We pre-trained a deep learning model on OCT images from the B&VIIT Eye Center (Seoul, South Korea) (N = 2602 eyes of 1301 patients). OCT images from Poona Eye Care (Pune, India) were chronologically sorted into adaptation training data (N = 60 eyes of 30 patients) for transfer learning and test data (N = 142 eyes of 71 patients) for validation. Deep learning models were trained to predict spherical equivalent (SE) and mean keratometry (K) values via transfer learning for domain adaptation.

RESULTS

Both adaptation models for SE and K were significantly better than those without adaptation (P < 0.001). In myopia/hyperopia classification, the model trained on circular optic disc OCT images yielded the best performance (accuracy = 74.7%). It also performed best in estimating SE with the lowest mean absolute error (MAE) of 1.58 D. For classifying the degree of corneal curvature, the optic nerve vertical algorithm performed best (accuracy = 65.7%). The optic nerve horizontal model achieved the lowest MAE (1.85 D) when predicting the K value. Saliency maps frequently highlighted the retinal nerve fiber layers.

CONCLUSIONS

Adaptation training via transfer learning is an effective technique for estimating refractive errors and K values using macular and optic nerve OCT images from ethnically heterogeneous populations. Further studies with larger sample sizes and various data sources are needed to confirm the feasibility of the proposed algorithm.

Risk of Bias in Chest Radiography Deep Learning Foundation Models

PURPOSE

To analyze a recently published chest radiography foundation model for the presence of biases that could lead to subgroup performance disparities across biologic sex and race.

MATERIALS AND METHODS

This Health Insurance Portability and Accountability Act-compliant retrospective study used 127 118 chest radiographs from 42 884 patients (mean age, 63 years ± 17 [SD]; 23 623 male, 19 261 female) from the CheXpert dataset that were collected between October 2002 and July 2017. To determine the presence of bias in features generated by a chest radiography foundation model and baseline deep learning model, dimensionality reduction methods together with two-sample Kolmogorov-Smirnov tests were used to detect distribution shifts across sex and race. A comprehensive disease detection performance analysis was then performed to associate any biases in the features to specific disparities in classification performance across patient subgroups.

RESULTS

Ten of 12 pairwise comparisons across biologic sex and race showed statistically significant differences in the studied foundation model, compared with four significant tests in the baseline model. Significant differences were found between male and female (P < .001) and Asian and Black (P < .001) patients in the feature projections that primarily capture disease. Compared with average model performance across all subgroups, classification performance on the "no finding" label decreased between 6.8% and 7.8% for female patients, and performance in detecting "pleural effusion" decreased between 10.7% and 11.6% for Black patients.

CONCLUSION

The studied chest radiography foundation model demonstrated racial and sex-related bias, which led to disparate performance across patient subgroups; thus, this model may be unsafe for clinical applications.Keywords: Conventional Radiography, Computer Application-Detection/Diagnosis, Chest Radiography, Bias, Foundation Models Supplemental material is available for this article. Published under a CC BY 4.0 license.See also commentary by Czum and Parr in this issue.

Efficiently Training Vision Transformers on Structural MRI Scans for Alzheimer's Disease Detection

Neuroimaging of large populations is valuable to identify factors that promote or resist brain disease, and to assist diagnosis, subtyping, and prognosis. Data-driven models such as convolutional neural networks (CNNs) have increasingly been applied to brain images to perform diagnostic and prognostic tasks by learning robust features. Vision transformers (ViT) - a new class of deep learning architectures - have emerged in recent years as an alternative to CNNs for several computer vision applications. Here we tested variants of the ViT architecture for a range of desired neuroimaging downstream tasks based on difficulty, in this case for sex and Alzheimer's disease (AD) classification based on 3D brain MRI. In our experiments, two vision transformer architecture variants achieved an AUC of 0.987 for sex and 0.892 for AD classification, respectively. We independently evaluated our models on data from two benchmark AD datasets. We achieved a performance boost of 5% and 9-10% upon fine-tuning vision transformer models pre-trained on synthetic (generated by a latent diffusion model) and real MRI scans, respectively. Our main contributions include testing the effects of different ViT training strategies including pre-training, data augmentation and learning rate warm-ups followed by annealing, as pertaining to the neuroimaging domain. These techniques are essential for training ViT-like models for neuroimaging applications where training data is usually limited. We also analyzed the effect of the amount of training data utilized on the test-time performance of the ViT via data-model scaling curves.Clinical Relevance- The models evaluated in this work could be trained on neuroimaging data to assist in diagnosis, subtyping and prognosis of Alzheimer's disease.

Is AI the way forward for reducing metal artifacts in CT? development of a generic deep learning-based method and initial evaluation in patients with sacroiliac joint implants

PURPOSE

To develop a deep learning-based metal artifact reduction technique (dl-MAR) and quantitatively compare metal artifacts on dl-MAR-corrected CT-images, orthopedic metal artifact reduction (O-MAR)-corrected CT-images and uncorrected CT-images after sacroiliac (SI) joint fusion.

METHODS

dl-MAR was trained on CT-images with simulated metal artifacts. Pre-surgery CT-images and uncorrected, O-MAR-corrected and dl-MAR-corrected post-surgery CT-images of twenty-five patients undergoing SI joint fusion were retrospectively obtained. Image registration was applied to align pre-surgery with post-surgery CT-images within each patient, allowing placement of regions of interest (ROIs) on the same anatomical locations. Six ROIs were placed on the metal implant and the contralateral side in bone lateral of the SI joint, the gluteus medius muscle and the iliacus muscle. Metal artifacts were quantified as the difference in Hounsfield units (HU) between pre- and post-surgery CT-values within the ROIs on the uncorrected, O-MAR-corrected and dl-MAR-corrected images. Noise was quantified as standard deviation in HU within the ROIs. Metal artifacts and noise in the post-surgery CT-images were compared using linear multilevel regression models.

RESULTS

Metal artifacts were significantly reduced by O-MAR and dl-MAR in bone (p < 0.001), contralateral bone (O-MAR: p = 0.009; dl-MAR: p < 0.001), gluteus medius (p < 0.001), contralateral gluteus medius (p < 0.001), iliacus (p < 0.001) and contralateral iliacus (O-MAR: p = 0.024; dl-MAR: p < 0.001) compared to uncorrected images. Images corrected with dl-MAR resulted in stronger artifact reduction than images corrected with O-MAR in contralateral bone (p < 0.001), gluteus medius (p = 0.006), contralateral gluteus medius (p < 0.001), iliacus (p = 0.017), and contralateral iliacus (p < 0.001). Noise was reduced by O-MAR in bone (p = 0.009) and gluteus medius (p < 0.001) while noise was reduced by dl-MAR in all ROIs (p < 0.001) in comparison to uncorrected images.

CONCLUSION

dl-MAR showed superior metal artifact reduction compared to O-MAR in CT-images with SI joint fusion implants.

Left Ventricle Detection from Cardiac Magnetic Resonance Relaxometry Images Using Visual Transformer

Left Ventricle (LV) detection from Cardiac Magnetic Resonance (CMR) imaging is a fundamental step, preliminary to myocardium segmentation and characterization. This paper focuses on the application of a Visual Transformer (ViT), a novel neural network architecture, to automatically detect LV from CMR relaxometry sequences. We implemented an object detector based on the ViT model to identify LV from CMR multi-echo T2* sequences. We evaluated performances differentiated by slice location according to the American Heart Association model using 5-fold cross-validation and on an independent dataset of CMR T2*, T2, and T1 acquisitions. To the best of our knowledge, this is the first attempt to localize LV from relaxometry sequences and the first application of ViT for LV detection. We collected an Intersection over Union (IoU) index of 0.68 and a Correct Identification Rate (CIR) of blood pool centroid of 0.99, comparable with other state-of-the-art methods. IoU and CIR values were significantly lower in apical slices. No significant differences in performances were assessed on independent T2* dataset (IoU = 0.68, p = 0.405; CIR = 0.94, p = 0.066). Performances were significantly worse on the T2 and T1 independent datasets (T2: IoU = 0.62, CIR = 0.95; T1: IoU = 0.67, CIR = 0.98), but still encouraging considering the different types of acquisition. This study confirms the feasibility of the application of ViT architectures in LV detection and defines a benchmark for relaxometry imaging.