Random forest regression for magnetic resonance image synthesis

Symmetric deformable registration of multimodal brain magnetic resonance images via appearance residuals

BACKGROUND AND OBJECTIVE

Deformable registration of multimodal brain magnetic resonance images presents significant challenges, primarily due to substantial structural variations between subjects and pronounced differences in appearance across imaging modalities.

METHODS

Here, we propose to symmetrically register images from two modalities based on appearance residuals from one modality to another. Computed with simple subtraction between modalities, the appearance residuals enhance structural details and form a common representation for simplifying multimodal deformable registration. The proposed framework consists of three serially connected modules: (i) an appearance residual module, which learns intensity residual maps between modalities with a cycle-consistent loss; (ii) a deformable registration module, which predicts deformations across subjects based on appearance residuals; and (iii) a deblurring module, which enhances the warped images to match the sharpness of the original images.

RESULTS

The proposed method, evaluated on two public datasets (HCP and LEMON), achieves the highest registration accuracy with topology preservation when compared with state-of-the-art methods.

CONCLUSIONS

Our residual space-guided registration framework, combined with GAN-based image enhancement, provides an effective solution to the challenges of multimodal deformable registration. By mitigating intensity distribution discrepancies and improving image quality, this approach improves registration accuracy and strengthens its potential for clinical application.

Interpretable machine learning models for the prediction of all-cause mortality and time to death in hemodialysis patients

INTRODUCTION

The elevated mortality and hospitalization rates among hemodialysis (HD) patients underscore the necessity for the development of accurate predictive tools. This study developed two models for predicting all-cause mortality and time to death-one using a comprehensive database and another simpler model based on demographic and clinical data without laboratory tests.

METHOD

A retrospective cohort study was conducted from January 2017 to June 2023. Two models were created: Model A with 85 variables and Model B with 22 variables. We assessed the models using random forest (RF), support vector machine, and logistic regression, comparing their performance via the AU-ROC. The RF regression model was used to predict time to death. To identify the most relevant factors for prediction, the Shapley value method was used.

RESULTS

Among 359 HD patients, the RF model provided the most reliable prediction. The optimized Model A showed an AU-ROC of 0.86 ± 0.07, a sensitivity of 0.86, and a specificity of 0.75 for predicting all-cause mortality. It also had an R2 of 0.59 for predicting time to death. The optimized Model B had an AU-ROC of 0.80 ± 0.06, a sensitivity of 0.81, and a specificity of 0.70 for predicting all-cause mortality. In addition, it had an R2 of 0.81 for predicting time to death.

CONCLUSION

Two new interpretable clinical tools have been proposed to predict all-cause mortality and time to death in HD patients using machine learning models. The minimal and readily accessible data on which Model B is based makes it a valuable tool for integrating into clinical decision-making processes.

Bridging MRI Cross-Modality Synthesis and Multi-Contrast Super-Resolution by Fine-Grained Difference Learning

In multi-modal magnetic resonance imaging (MRI), the tasks of imputing or reconstructing the target modality share a common obstacle: the accurate modeling of fine-grained inter-modal differences, which has been sparingly addressed in current literature. These differences stem from two sources: 1) spatial misalignment remaining after coarse registration and 2) structural distinction arising from modality-specific signal manifestations. This paper integrates the previously separate research trajectories of cross-modality synthesis (CMS) and multi-contrast super-resolution (MCSR) to address this pervasive challenge within a unified framework. Connected through generalized down-sampling ratios, this unification not only emphasizes their common goal in reducing structural differences, but also identifies the key task distinguishing MCSR from CMS: modeling the structural distinctions using the limited information from the misaligned target input. Specifically, we propose a composite network architecture with several key components: a label correction module to align the coordinates of multi-modal training pairs, a CMS module serving as the base model, an SR branch to handle target inputs, and a difference projection discriminator for structural distinction-centered adversarial training. When training the SR branch as the generator, the adversarial learning is enhanced with distinction-aware incremental modulation to ensure better-controlled generation. Moreover, the SR branch integrates deformable convolutions to address cross-modal spatial misalignment at the feature level. Experiments conducted on three public datasets demonstrate that our approach effectively balances structural accuracy and realism, exhibiting overall superiority in comprehensive evaluations for both tasks over current state-of-the-art approaches. The code is available at https://github.com/papshare/FGDL.

Unified Multi-Modal Image Synthesis for Missing Modality Imputation

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction

Multi-modal magnetic resonance imaging (MRI) plays a crucial role in comprehensive disease diagnosis in clinical medicine. However, acquiring certain modalities, such as T2-weighted images (T2WIs), is time-consuming and prone to be with motion artifacts. It negatively impacts subsequent multi-modal image analysis. To address this issue, we propose an end-to-end deep learning framework that utilizes T1-weighted images (T1WIs) as auxiliary modalities to expedite T2WIs' acquisitions. While image pre-processing is capable of mitigating misalignment, improper parameter selection leads to adverse pre-processing effects, requiring iterative experimentation and adjustment. To overcome this shortage, we employ Optimal Transport (OT) to synthesize T2WIs by aligning T1WIs and performing cross-modal synthesis, effectively mitigating spatial misalignment effects. Furthermore, we adopt an alternating iteration framework between the reconstruction task and the cross-modal synthesis task to optimize the final results. Then, we prove that the reconstructed T2WIs and the synthetic T2WIs become closer on the T2 image manifold with iterations increasing, and further illustrate that the improved reconstruction result enhances the synthesis process, whereas the enhanced synthesis result improves the reconstruction process. Finally, experimental results from FastMRI and internal datasets confirm the effectiveness of our method, demonstrating significant improvements in image reconstruction quality even at low sampling rates.

Reliable multi-modal medical image-to-image translation independent of pixel-wise aligned data

BACKGROUND

The current mainstream multi-modal medical image-to-image translation methods face a contradiction. Supervised methods with outstanding performance rely on pixel-wise aligned training data to constrain the model optimization. However, obtaining pixel-wise aligned multi-modal medical image datasets is challenging. Unsupervised methods can be trained without paired data, but their reliability cannot be guaranteed. At present, there is no ideal multi-modal medical image-to-image translation method that can generate reliable translation results without the need for pixel-wise aligned data.

PURPOSE

This work aims to develop a novel medical image-to-image translation model that is independent of pixel-wise aligned data (MITIA), enabling reliable multi-modal medical image-to-image translation under the condition of misaligned training data.

METHODS

The proposed MITIA model utilizes a prior extraction network composed of a multi-modal medical image registration module and a multi-modal misalignment error detection module to extract pixel-level prior information from training data with misalignment errors to the largest extent. The extracted prior information is then used to construct a regularization term to constrain the optimization of the unsupervised cycle-consistent Generative Adversarial Network model, restricting its solution space and thereby improving the performance and reliability of the generator. We trained the MITIA model using six datasets containing different misalignment errors and two well-aligned datasets. Subsequently, we conducted quantitative analysis using peak signal-to-noise ratio and structural similarity as metrics. Moreover, we compared the proposed method with six other state-of-the-art image-to-image translation methods.

RESULTS

The results of both quantitative analysis and qualitative visual inspection indicate that MITIA achieves superior performance compared to the competing state-of-the-art methods, both on misaligned data and aligned data. Furthermore, MITIA shows more stability in the presence of misalignment errors in the training data, regardless of their severity or type.

CONCLUSIONS

The proposed method achieves outstanding performance in multi-modal medical image-to-image translation tasks without aligned training data. Due to the difficulty in obtaining pixel-wise aligned data for medical image translation tasks, MITIA is expected to generate significant application value in this scenario compared to existing methods.

Reducing Gadolinium Contrast With Artificial Intelligence

Gadolinium contrast is an important agent in magnetic resonance imaging (MRI), particularly in neuroimaging where it can help identify blood-brain barrier breakdown from an inflammatory, infectious, or neoplastic process. However, gadolinium contrast has several drawbacks, including nephrogenic systemic fibrosis, gadolinium deposition in the brain and bones, and allergic-like reactions. As computer hardware and technology continues to evolve, machine learning has become a possible solution for eliminating or reducing the dose of gadolinium contrast. This review summarizes the clinical uses of gadolinium contrast, the risks of gadolinium contrast, and state-of-the-art machine learning methods that have been applied to reduce or eliminate gadolinium contrast administration, as well as their current limitations, with a focus on neuroimaging applications. EVIDENCE LEVEL: 3 TECHNICAL EFFICACY: Stage 1.

Deep learning for the harmonization of structural MRI scans: a survey

Medical imaging datasets for research are frequently collected from multiple imaging centers using different scanners, protocols, and settings. These variations affect data consistency and compatibility across different sources. Image harmonization is a critical step to mitigate the effects of factors like inherent differences between various vendors, hardware upgrades, protocol changes, and scanner calibration drift, as well as to ensure consistent data for medical image processing techniques. Given the critical importance and widespread relevance of this issue, a vast array of image harmonization methodologies have emerged, with deep learning-based approaches driving substantial advancements in recent times. The goal of this review paper is to examine the latest deep learning techniques employed for image harmonization by analyzing cutting-edge architectural approaches in the field of medical image harmonization, evaluating both their strengths and limitations. This paper begins by providing a comprehensive fundamental overview of image harmonization strategies, covering three critical aspects: established imaging datasets, commonly used evaluation metrics, and characteristics of different scanners. Subsequently, this paper analyzes recent structural MRI (Magnetic Resonance Imaging) harmonization techniques based on network architecture, network learning algorithm, network supervision strategy, and network output. The underlying architectures include U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based generative models, transformer-based approaches, as well as custom-designed network architectures. This paper investigates the effectiveness of Disentangled Representation Learning (DRL) as a pivotal learning algorithm in harmonization. Lastly, the review highlights the primary limitations in harmonization techniques, specifically the lack of comprehensive quantitative comparisons across different methods. The overall aim of this review is to serve as a guide for researchers and practitioners to select appropriate architectures based on their specific conditions and requirements. It also aims to foster discussions around ongoing challenges in the field and shed light on promising future research directions with the potential for significant advancements.

Using artificial intelligence to rapidly identify microplastics pollution and predict microplastics environmental behaviors

With the massive release of microplastics (MPs) into the environment, research related to MPs is advancing rapidly. Effective research methods are necessary to identify the chemical composition, shape, distribution, and environmental impacts of MPs. In recent years, artificial intelligence (AI)-driven machine learning methods have demonstrated excellent performance in analyzing MPs in soil and water. This review provides a comprehensive overview of machine learning methods for the prediction of MPs for various tasks, and discusses in detail the data source, data preprocessing, algorithm principle, and algorithm limitation of applied machine learning. In addition, this review discusses the limitation of current machine learning methods for various task analysis in MPs along with future prospect. Finally, this review finds research potential in future work in building large generalized MPs datasets, designing high-performance but low-computational-complexity algorithms, and evaluating model interpretability.

Conversion of T2-Weighted Magnetic Resonance Images of Cervical Spine Trauma to Short T1 Inversion Recovery (STIR) Images by Generative Adversarial Network

INTRODUCTION

The short T1 inversion recovery (STIR) sequence is advantageous for visualizing ligamentous injuries, but the STIR sequence may be missing in some cases. The purpose of this study was to generate synthetic STIR images from MRI T2-weighted images (T2WI) of patients with cervical spine trauma using a generative adversarial network (GAN).  Methods: A total of 969 pairs of T2WI and STIR images were extracted from 79 patients with cervical spine trauma. The synthetic model was trained 100 times, and the performance of the model was evaluated with five-fold cross-validation.  Results: As for quantitative validation, the structural similarity score was 0.519±0.1 and the peak signal-to-noise ratio score was 19.37±1.9 dB. As for qualitative validation, the incorporation of synthetic STIR images generated by a GAN alongside T2WI substantially enhances sensitivity in the detection of interspinous ligament injuries, outperforming assessments reliant solely on T2WI.

CONCLUSION

The GAN model can generate synthetic STIRs from T2 images of cervical spine trauma using image-to-image conversion techniques. The use of a combination of synthetic STIR images generated by a GAN and T2WI improves sensitivity in detecting interspinous ligament injuries compared to assessments that use only T2WI.

One model to unite them all: Personalized federated learning of multi-contrast MRI synthesis

Curation of large, diverse MRI datasets via multi-institutional collaborations can help improve learning of generalizable synthesis models that reliably translate source- onto target-contrast images. To facilitate collaborations, federated learning (FL) adopts decentralized model training while mitigating privacy concerns by avoiding sharing of imaging data. However, conventional FL methods can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident within and across imaging sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) that improves reliability against data heterogeneity via model specialization to individual sites and synthesis tasks (i.e., source-target contrasts). To do this, pFLSynth leverages an adversarial model equipped with novel personalization blocks that control the statistics of generated feature maps across the spatial/channel dimensions, given latent variables specific to sites and tasks. To further promote communication efficiency and site specialization, partial network aggregation is employed over later generator stages while earlier generator stages and the discriminator are trained locally. As such, pFLSynth enables multi-task training of multi-site synthesis models with high generalization performance across sites and tasks. Comprehensive experiments demonstrate the superior performance and reliability of pFLSynth in MRI synthesis against prior federated methods.

Machine learning insights into PM2.5 changes during COVID-19 lockdown: LSTM and RF analysis in Mashhad

This study seeks to investigate the impact of COVID-19 lockdown measures on air quality in the city of Mashhad employing two strategies. We initiated our research using basic statistical methods such as paired sample t-tests to compare hourly PM2.5 data in two scenarios: before and during quarantine, and pre- and post-lockdown. This initial analysis provided a broad understanding of potential changes in air quality. Notably, a low reduction of 2.40% in PM2.5 was recorded when compared to air quality prior to the lockdown period. This finding highlights the wide range of factors that impact the levels of particulate matter in urban settings, with the transportation sector often being widely recognized as one of the principal causes of this issue. Nevertheless, throughout the period after the quarantine, a remarkable decrease in air quality was observed characterized by distinct seasonal patterns, in contrast to previous years. This finding demonstrates a significant correlation between changes in human mobility patterns and their influence on the air quality of urban areas. It also emphasizes the need to use air pollution modeling as a fundamental tool to evaluate and understand these linkages to support long-term plans for reducing air pollution. To obtain a more quantitative understanding, we then employed cutting-edge machine learning methods, such as random forest and long short-term memory algorithms, to accurately determine the effect of the lockdown on PM2.5 levels. Our models' results demonstrated remarkable efficacy in assessing the pollutant concentration in Mashhad during lockdown measures. The test set yielded an R-squared value of 0.82 for the long short-term memory network model, whereas the random forest model showed a calculated cross-validation R-squared of 0.78. The required computational cost for training the LSTM and the RF models across all data was 25 min and 3 s, respectively. In summary, through the integration of statistical methods and machine learning, this research attempts to provide a comprehensive understanding of the impact of human interventions on air quality dynamics.

Energy-guided diffusion model for CBCT-to-CT synthesis

Cone Beam Computed Tomography (CBCT) plays a crucial role in Image-Guided Radiation Therapy (IGRT), providing essential assurance of accuracy in radiation treatment by monitoring changes in anatomical structures during the treatment process. However, CBCT images often face interference from scatter noise and artifacts, posing a significant challenge when relying solely on CBCT for precise dose calculation and accurate tissue localization. There is an urgent need to enhance the quality of CBCT images, enabling a more practical application in IGRT. This study introduces EGDiff, a novel framework based on the diffusion model, designed to address the challenges posed by scatter noise and artifacts in CBCT images. In our approach, we employ a forward diffusion process by adding Gaussian noise to CT images, followed by a reverse denoising process using ResUNet with an attention mechanism to predict noise intensity, ultimately synthesizing CBCT-to-CT images. Additionally, we design an energy-guided function to retain domain-independent features and discard domain-specific features during the denoising process, enhancing the effectiveness of CBCT-CT generation. We conduct numerous experiments on the thorax dataset and pancreas dataset. The results demonstrate that EGDiff performs better on the thoracic tumor dataset with SSIM of 0.850, MAE of 26.87 HU, PSNR of 19.83 dB, and NCC of 0.874. EGDiff outperforms SoTA CBCT-to-CT synthesis methods on the pancreas dataset with SSIM of 0.754, MAE of 32.19 HU, PSNR of 19.35 dB, and NCC of 0.846. By improving the accuracy and reliability of CBCT images, EGDiff can enhance the precision of radiation therapy, minimize radiation exposure to healthy tissues, and ultimately contribute to more effective and personalized cancer treatment strategies.

Image harmonization improves consistency of intra-rater delineations of MS lesions in heterogeneous MRI

Clinical magnetic resonance images (MRIs) lack a standard intensity scale due to differences in scanner hardware and the pulse sequences used to acquire the images. When MRIs are used for quantification, as in the evaluation of white matter lesions (WMLs) in multiple sclerosis, this lack of intensity standardization becomes a critical problem affecting both the staging and tracking of the disease and its treatment. This paper presents a study of harmonization on WML segmentation consistency, which is evaluated using an object detection classification scheme that incorporates manual delineations from both the original and harmonized MRIs. A cohort of ten people scanned on two different imaging platforms was studied. An expert rater, blinded to the image source, manually delineated WMLs on images from both scanners before and after harmonization. It was found that there is closer agreement in both global and per-lesion WML volume and spatial distribution after harmonization, demonstrating the importance of image harmonization prior to the creation of manual delineations. These results could lead to better truth models in both the development and evaluation of automated lesion segmentation algorithms.

Learning multi-site harmonization of magnetic resonance images without traveling human phantoms

Harmonization improves Magn. Reson. Imaging (MRI) data consistency and is central to effective integration of diverse imaging data acquired across multiple sites. Recent deep learning techniques for harmonization are predominantly supervised in nature and hence require imaging data of the same human subjects to be acquired at multiple sites. Data collection as such requires the human subjects to travel across sites and is hence challenging, costly, and impractical, more so when sufficient sample size is needed for reliable network training. Here we show how harmonization can be achieved with a deep neural network that does not rely on traveling human phantom data. Our method disentangles site-specific appearance information and site-invariant anatomical information from images acquired at multiple sites and then employs the disentangled information to generate the image of each subject for any target site. We demonstrate with more than 6,000 multi-site T1- and T2-weighted images that our method is remarkably effective in generating images with realistic site-specific appearances without altering anatomical details. Our method allows retrospective harmonization of data in a wide range of existing modern large-scale imaging studies, conducted via different scanners and protocols, without additional data collection.

Unsupervised Medical Image Translation With Adversarial Diffusion Models

Imputation of missing images via source-to-target modality translation can improve diversity in medical imaging protocols. A pervasive approach for synthesizing target images involves one-shot mapping through generative adversarial networks (GAN). Yet, GAN models that implicitly characterize the image distribution can suffer from limited sample fidelity. Here, we propose a novel method based on adversarial diffusion modeling, SynDiff, for improved performance in medical image translation. To capture a direct correlate of the image distribution, SynDiff leverages a conditional diffusion process that progressively maps noise and source images onto the target image. For fast and accurate image sampling during inference, large diffusion steps are taken with adversarial projections in the reverse diffusion direction. To enable training on unpaired datasets, a cycle-consistent architecture is devised with coupled diffusive and non-diffusive modules that bilaterally translate between two modalities. Extensive assessments are reported on the utility of SynDiff against competing GAN and diffusion models in multi-contrast MRI and MRI-CT translation. Our demonstrations indicate that SynDiff offers quantitatively and qualitatively superior performance against competing baselines.

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities

Accurate segmentation of brain tumors is of critical importance in clinical assessment and treatment planning, which requires multiple MR modalities providing complementary information. However, due to practical limits, one or more modalities may be missing in real scenarios. To tackle this problem, existing methods need to train multiple networks or a unified but fixed network for various possible missing modality cases, which leads to high computational burdens or sub-optimal performance. In this paper, we propose a unified and adaptive multi-modal MR image synthesis method, and further apply it to tumor segmentation with missing modalities. Based on the decomposition of multi-modal MR images into common and modality-specific features, we design a shared hyper-encoder for embedding each available modality into the feature space, a graph-attention-based fusion block to aggregate the features of available modalities to the fused features, and a shared hyper-decoder for image reconstruction. We also propose an adversarial common feature constraint to enforce the fused features to be in a common space. As for missing modality segmentation, we first conduct the feature-level and image-level completion using our synthesis method and then segment the tumors based on the completed MR images together with the extracted common features. Moreover, we design a hypernet-based modulation module to adaptively utilize the real and synthetic modalities. Experimental results suggest that our method can not only synthesize reasonable multi-modal MR images, but also achieve state-of-the-art performance on brain tumor segmentation with missing modalities.

oFVSD: a Python package of optimized forward variable selection decoder for high-dimensional neuroimaging data

The complexity and high dimensionality of neuroimaging data pose problems for decoding information with machine learning (ML) models because the number of features is often much larger than the number of observations. Feature selection is one of the crucial steps for determining meaningful target features in decoding; however, optimizing the feature selection from such high-dimensional neuroimaging data has been challenging using conventional ML models. Here, we introduce an efficient and high-performance decoding package incorporating a forward variable selection (FVS) algorithm and hyper-parameter optimization that automatically identifies the best feature pairs for both classification and regression models, where a total of 18 ML models are implemented by default. First, the FVS algorithm evaluates the goodness-of-fit across different models using the k-fold cross-validation step that identifies the best subset of features based on a predefined criterion for each model. Next, the hyperparameters of each ML model are optimized at each forward iteration. Final outputs highlight an optimized number of selected features (brain regions of interest) for each model with its accuracy. Furthermore, the toolbox can be executed in a parallel environment for efficient computation on a typical personal computer. With the optimized forward variable selection decoder (oFVSD) pipeline, we verified the effectiveness of decoding sex classification and age range regression on 1,113 structural magnetic resonance imaging (MRI) datasets. Compared to ML models without the FVS algorithm and with the Boruta algorithm as a variable selection counterpart, we demonstrate that the oFVSD significantly outperformed across all of the ML models over the counterpart models without FVS (approximately 0.20 increase in correlation coefficient, r, with regression models and 8% increase in classification models on average) and with Boruta variable selection algorithm (approximately 0.07 improvement in regression and 4% in classification models). Furthermore, we confirmed the use of parallel computation considerably reduced the computational burden for the high-dimensional MRI data. Altogether, the oFVSD toolbox efficiently and effectively improves the performance of both classification and regression ML models, providing a use case example on MRI datasets. With its flexibility, oFVSD has the potential for many other modalities in neuroimaging. This open-source and freely available Python package makes it a valuable toolbox for research communities seeking improved decoding accuracy.

Joint learning-based feature reconstruction and enhanced network for incomplete multi-modal brain tumor segmentation

Multimodal Magnetic Resonance Imaging (MRI) can provide valuable complementary information and substantially enhance the performance of brain tumor segmentation. However, it is common for certain modalities to be absent or missing during clinical diagnosis, which can significantly impair segmentation techniques that rely on complete modalities. Current advanced methods attempt to address this challenge by developing shared feature representations via modal fusion to handle different missing modality situations. Considering the importance of missing modality information in multimodal segmentation, this paper utilize a feature reconstruction method to recover the missing information, and proposes a joint learning-based feature reconstruction and enhancement method for incomplete modality brain tumor segmentation. The method leverages an information learning mechanism to transfer information from the complete modality to a single modality, enabling it to obtain complete brain tumor information, even without the support of other modalities. Additionally, the method incorporates a module for reconstructing missing modality features, which recovers fused features of the absent modality through utilizing the abundant potential information obtained from the available modalities. Furthermore, the feature enhancement mechanism improves shared feature representation by utilizing the information obtained from the missing modalities that have been reconstructed. These processes enable the method to obtain more comprehensive information regarding brain tumors in various missing modality circumstances, thereby enhancing the model's robustness. The performance of the proposed model was evaluated on BraTS datasets and compared with other deep learning algorithms using Dice similarity scores. On the BraTS2018 dataset, the proposed algorithm achieved a Dice similarity score of 86.28%, 77.02%, and 59.64% for whole tumors, tumor cores, and enhanced tumors, respectively. These results demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations.

TISS-net: Brain tumor image synthesis and segmentation using cascaded dual-task networks and error-prediction consistency

Accurate segmentation of brain tumors from medical images is important for diagnosis and treatment planning, and it often requires multi-modal or contrast-enhanced images. However, in practice some modalities of a patient may be absent. Synthesizing the missing modality has a potential for filling this gap and achieving high segmentation performance. Existing methods often treat the synthesis and segmentation tasks separately or consider them jointly but without effective regularization of the complex joint model, leading to limited performance. We propose a novel brain Tumor Image Synthesis and Segmentation network (TISS-Net) that obtains the synthesized target modality and segmentation of brain tumors end-to-end with high performance. First, we propose a dual-task-regularized generator that simultaneously obtains a synthesized target modality and a coarse segmentation, which leverages a tumor-aware synthesis loss with perceptibility regularization to minimize the high-level semantic domain gap between synthesized and real target modalities. Based on the synthesized image and the coarse segmentation, we further propose a dual-task segmentor that predicts a refined segmentation and error in the coarse segmentation simultaneously, where a consistency between these two predictions is introduced for regularization. Our TISS-Net was validated with two applications: synthesizing FLAIR images for whole glioma segmentation, and synthesizing contrast-enhanced T1 images for Vestibular Schwannoma segmentation. Experimental results showed that our TISS-Net largely improved the segmentation accuracy compared with direct segmentation from the available modalities, and it outperformed state-of-the-art image synthesis-based segmentation methods.

Exploring the Possibility of Measuring Vertebrae Bone Structure Metrics Using MDCT Images: An Unpaired Image-to-Image Translation Method

Bone structure metrics are vital for the evaluation of vertebral bone strength. However, the gold standard for measuring bone structure metrics, micro-Computed Tomography (micro-CT), cannot be used in vivo, which hinders the early diagnosis of fragility fractures. This paper used an unpaired image-to-image translation method to capture the mapping between clinical multidetector computed tomography (MDCT) and micro-CT images and then generated micro-CT-like images to measure bone structure metrics. MDCT and micro-CT images were scanned from 75 human lumbar spine specimens and formed training and testing sets. The generator in the model focused on learning both the structure and detailed pattern of bone trabeculae and generating micro-CT-like images, and the discriminator determined whether the generated images were micro-CT images or not. Based on similarity metrics (i.e., SSIM and FID) and bone structure metrics (i.e., bone volume fraction, trabecular separation and trabecular thickness), a set of comparisons were performed. The results show that the proposed method can perform better in terms of both similarity metrics and bone structure metrics and the improvement is statistically significant. In particular, we compared the proposed method with the paired image-to-image method and analyzed the pros and cons of the method used.

Virtual high-resolution MR angiography from non-angiographic multi-contrast MRIs: synthetic vascular model populations for in-silico trials

Despite success on multi-contrast MR image synthesis, generating specific modalities remains challenging. Those include Magnetic Resonance Angiography (MRA) that highlights details of vascular anatomy using specialised imaging sequences for emphasising inflow effect. This work proposes an end-to-end generative adversarial network that can synthesise anatomically plausible, high-resolution 3D MRA images using commonly acquired multi-contrast MR images (e.g. T1/T2/PD-weighted MR images) for the same subject whilst preserving the continuity of vascular anatomy. A reliable technique for MRA synthesis would unleash the research potential of very few population databases with imaging modalities (such as MRA) that enable quantitative characterisation of whole-brain vasculature. Our work is motivated by the need to generate digital twins and virtual patients of cerebrovascular anatomy for in-silico studies and/or in-silico trials. We propose a dedicated generator and discriminator that leverage the shared and complementary features of multi-source images. We design a composite loss function for emphasising vascular properties by minimising the statistical difference between the feature representations of the target images and the synthesised outputs in both 3D volumetric and 2D projection domains. Experimental results show that the proposed method can synthesise high-quality MRA images and outperform the state-of-the-art generative models both qualitatively and quantitatively. The importance assessment reveals that T2 and PD-weighted images are better predictors of MRA images than T1; and PD-weighted images contribute to better visibility of small vessel branches towards the peripheral regions. In addition, the proposed approach can generalise to unseen data acquired at different imaging centres with different scanners, whilst synthesising MRAs and vascular geometries that maintain vessel continuity. The results show the potential for use of the proposed approach to generating digital twin cohorts of cerebrovascular anatomy at scale from structural MR images typically acquired in population imaging initiatives.

MRI image synthesis for fluid-attenuated inversion recovery and diffusion-weighted images with deep learning

This study aims to synthesize fluid-attenuated inversion recovery (FLAIR) and diffusion-weighted images (DWI) with a deep conditional adversarial network from T1- and T2-weighted magnetic resonance imaging (MRI) images. A total of 1980 images of 102 patients were split into two datasets: 1470 (68 patients) in a training set and 510 (34 patients) in a test set. The prediction framework was based on a convolutional neural network with a generator and discriminator. T1-weighted, T2-weighted, and composite images were used as inputs. The digital imaging and communications in medicine (DICOM) images were converted to 8-bit red-green-blue images. The red and blue channels of the composite images were assigned to 8-bit grayscale pixel values in T1-weighted images, and the green channel was assigned to those in T2-weighted images. The prediction FLAIR and DWI images were of the same objects as the inputs. For the results, the prediction model with composite MRI input images in the DWI image showed the smallest relative mean absolute error (rMAE) and largest mutual information (MI), and that in the FLAIR image showed the largest relative mean-square error (rMSE), relative root-mean-square error (rRMSE), and peak signal-to-noise ratio (PSNR). For the FLAIR image, the prediction model with the T2-weighted MRI input images generated more accurate synthesis results than that with the T1-weighted inputs. The proposed image synthesis framework can improve the versatility and quality of multi-contrast MRI without extra scans. The composite input MRI image contributes to synthesizing the multi-contrast MRI image efficiently.

Developing and evaluating a machine-learning-based algorithm to predict the incidence and severity of ARDS with continuous non-invasive parameters from ordinary monitors and ventilators

OBJECTIVES

Major observational studies report that the mortality rate of acute respiratory distress syndrome (ARDS) is close to 40%. Different treatment strategies are required for each patient, according to the degree of ARDS. Early prediction of ARDS is helpful to implement targeted drug therapy and mechanical ventilation strategies for patients with different degrees of potential ARDS. In this paper, a new dynamic prediction machine learning model for ARDS incidence and severity is established and evaluated based on 28 parameters from ordinary monitors and ventilators, capable of dynamic prediction of the incidence and severity of ARDS. This new method is expected to meet the clinical practice requirements of user-friendliness and timeliness for wider application.

METHODS

A total of 4738 hospitalized patients who required ICU care from 159 hospitals are employed in this study. The models are trained by standardized data from electronic medical records. There are 28 structured, continuous non-invasive parameters that are recorded every hour. Seven machine learning models using only continuous, non-invasive parameters are developed for dynamic prediction and compared with methods trained by complete parameters and the traditional risk adjustment method (i.e., oxygenation saturation index method).

RESULTS

The optimal prediction performance (area under the curve) of the ARDS incidence and severity prediction models built using continuous noninvasive parameters reached0.8691 and 0.7765, respectively. In terms of mild and severe ARDS prediction, the AUC values are both above 0.85. The performance of the model using only continuous non-invasive parameters have an AUC of 0.0133 lower, in comparison with that employing a complete feature set, including continuous non-invasive parameters, demographic information, laboratory parameters and clinical natural language text.

CONCLUSIONS

A machine learning method was developed in this study using only continuous non-invasive parameters for ARDS incidence and severity prediction. Because the continuous non-invasive parameters can be easily obtained from ordinary monitors and ventilators, the method presented in this study is friendly and convenient to use. It is expected to be applied in pre-hospital setting for early ARDS warning.

A literature survey of MR-based brain tumor segmentation with missing modalities

Multimodal MR brain tumor segmentation is one of the hottest issues in the community of medical image processing. However, acquiring the complete set of MR modalities is not always possible in clinical practice, due to the acquisition protocols, image corruption, scanner availability, scanning cost or allergies to certain contrast materials. The missing information can cause some restraints to brain tumor diagnosis, monitoring, treatment planning and prognosis. Thus, it is highly desirable to develop brain tumor segmentation methods to address the missing modalities problem. Based on the recent advancements, in this review, we provide a detailed analysis of the missing modality issue in MR-based brain tumor segmentation. First, we briefly introduce the biomedical background concerning brain tumor, MR imaging techniques, and the current challenges in brain tumor segmentation. Then, we provide a taxonomy of the state-of-the-art methods with five categories, namely, image synthesis-based method, latent feature space-based model, multi-source correlation-based method, knowledge distillation-based method, and domain adaptation-based method. In addition, the principles, architectures, benefits and limitations are elaborated in each method. Following that, the corresponding datasets and widely used evaluation metrics are described. Finally, we analyze the current challenges and provide a prospect for future development trends. This review aims to provide readers with a thorough knowledge of the recent contributions in the field of brain tumor segmentation with missing modalities and suggest potential future directions.

An Efficient Optimization Approach for Glioma Tumor Segmentation in Brain MRI

Glioma is an aggressive type of cancer that develops in the brain or spinal cord. Due to many differences in its shape and appearance, accurate segmentation of glioma for identifying all parts of the tumor and its surrounding cancerous tissues is a challenging task. In recent researches, the combination of multi-atlas segmentation and machine learning methods provides robust and accurate results by learning from annotated atlas datasets. To overcome the side effects of limited existing information on atlas-based segmentation, and the long training phase of learning methods, we proposed a semi-supervised unified framework for multi-label segmentation that formulates this problem in terms of a Markov Random Field energy optimization on a parametric graph. To evaluate the proposed framework, we apply it to publicly available BRATS datasets, including low- and high-grade glioma tumors. Experimental results indicate competitive performance compared to the state-of-the-art methods. Compared with the top ranked methods, the proposed framework obtains the best dice score for segmenting of "whole tumor" (WT), "tumor core" (TC ) and "enhancing active tumor" (ET) regions. The achieved accuracy is 94[Formula: see text] characterized by the mean dice score. The motivation of using MRF graph is to map the segmentation problem to an optimization model in a graphical environment. Therefore, by defining perfect graph structure and optimum constraints and flows in the continuous max-flow model, the segmentation is performed precisely.

ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis

Generative adversarial models with convolutional neural network (CNN) backbones have recently been established as state-of-the-art in numerous medical image synthesis tasks. However, CNNs are designed to perform local processing with compact filters, and this inductive bias compromises learning of contextual features. Here, we propose a novel generative adversarial approach for medical image synthesis, ResViT, that leverages the contextual sensitivity of vision transformers along with the precision of convolution operators and realism of adversarial learning. ResViT's generator employs a central bottleneck comprising novel aggregated residual transformer (ART) blocks that synergistically combine residual convolutional and transformer modules. Residual connections in ART blocks promote diversity in captured representations, while a channel compression module distills task-relevant information. A weight sharing strategy is introduced among ART blocks to mitigate computational burden. A unified implementation is introduced to avoid the need to rebuild separate synthesis models for varying source-target modality configurations. Comprehensive demonstrations are performed for synthesizing missing sequences in multi-contrast MRI, and CT images from MRI. Our results indicate superiority of ResViT against competing CNN- and transformer-based methods in terms of qualitative observations and quantitative metrics.

PTNet3D: A 3D High-Resolution Longitudinal Infant Brain MRI Synthesizer Based on Transformers

An increased interest in longitudinal neurodevelopment during the first few years after birth has emerged in recent years. Noninvasive magnetic resonance imaging (MRI) can provide crucial information about the development of brain structures in the early months of life. Despite the success of MRI collections and analysis for adults, it remains a challenge for researchers to collect high-quality multimodal MRIs from developing infant brains because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still during scanning. In addition, there are limited analytic approaches available. These challenges often lead to a significant reduction of usable MRI scans and pose a problem for modeling neurodevelopmental trajectories. Researchers have explored solving this problem by synthesizing realistic MRIs to replace corrupted ones. Among synthesis methods, the convolutional neural network-based (CNN-based) generative adversarial networks (GANs) have demonstrated promising performance. In this study, we introduced a novel 3D MRI synthesis framework- pyramid transformer network (PTNet3D)- which relies on attention mechanisms through transformer and performer layers. We conducted extensive experiments on high-resolution Developing Human Connectome Project (dHCP) and longitudinal Baby Connectome Project (BCP) datasets. Compared with CNN-based GANs, PTNet3D consistently shows superior synthesis accuracy and superior generalization on two independent, large-scale infant brain MRI datasets. Notably, we demonstrate that PTNet3D synthesized more realistic scans than CNN-based models when the input is from multi-age subjects. Potential applications of PTNet3D include synthesizing corrupted or missing images. By replacing corrupted scans with synthesized ones, we observed significant improvement in infant whole brain segmentation.

Evaluating the use of synthetic T1-w images in new T2 lesion detection in multiple sclerosis

The assessment of disease activity using serial brain MRI scans is one of the most valuable strategies for monitoring treatment response in patients with multiple sclerosis (MS) receiving disease-modifying treatments. Recently, several deep learning approaches have been proposed to improve this analysis, obtaining a good trade-off between sensitivity and specificity, especially when using T1-w and T2-FLAIR images as inputs. However, the need to acquire two different types of images is time-consuming, costly and not always available in clinical practice. In this paper, we investigate an approach to generate synthetic T1-w images from T2-FLAIR images and subsequently analyse the impact of using original and synthetic T1-w images on the performance of a state-of-the-art approach for longitudinal MS lesion detection. We evaluate our approach on a dataset containing 136 images from MS patients, and 73 images with lesion activity (the appearance of new T2 lesions in follow-up scans). To evaluate the synthesis of the images, we analyse the structural similarity index metric and the median absolute error and obtain consistent results. To study the impact of synthetic T1-w images, we evaluate the performance of the new lesion detection approach when using (1) both T2-FLAIR and T1-w original images, (2) only T2-FLAIR images, and (3) both T2-FLAIR and synthetic T1-w images. Sensitivities of 0.75, 0.63, and 0.81, respectively, were obtained at the same false-positive rate (0.14) for all experiments. In addition, we also present the results obtained when using the data from the international MSSEG-2 challenge, showing also an improvement when including synthetic T1-w images. In conclusion, we show that the use of synthetic images can support the lack of data or even be used instead of the original image to homogenize the contrast of the different acquisitions in new T2 lesions detection algorithms.

Machine learning-enabled optimization of extrusion-based 3D printing

Machine learning (ML) and three-dimensional (3D) printing are among the fastest-growing branches of science. While ML can enable computers to independently learn from available data to make decisions with minimal human intervention, 3D printing has opened up an avenue for modern, multi-material, manufacture of complex 3D structures with a rapid turn-around ability for users with limited manufacturing experience. However, the determination of optimum printing parameters is still a challenge, increasing pre-printing process time and material wastage. Here, we present the first integration of ML and 3D printing through an easy-to-use graphical user interface (GUI) for printing parameter optimization. Unlike the widely held orthogonal design used in most of the 3D printing research, we, for the first time, used nine different computer-aided design (CAD) images and in order to enable ML algorithms to distinguish the difference between designs, we devised a self-designed method to calculate the "complexity index" of CAD designs. In addition, for the first time, the similarity of the print outcomes and CAD images are measured using four different self-designed labeling methods (both manually and automatically) to figure out the best labeling method for ML purposes. Subsequently, we trained eight ML algorithms on 224 datapoints to identify the best ML model for 3D printing applications. The "gradient boosting regression" model yields the best prediction performance with an R-2 score of 0.954. The ML-embedded GUI developed in this study enables users (either skilled or unskilled in 3D printing and/or ML) to simply upload a design (desired to print) to the GUI along with desired printing temperature and pressure to obtain the approximate similarity in the case of actual 3D printing of the uploaded design. This ultimately can prevent error-and-trial steps prior to printing which in return can speed up overall design-to-end-product time with less material waste and more cost-efficiency.

Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions

Removing the bias and variance of multicentre data has always been a challenge in large scale digital healthcare studies, which requires the ability to integrate clinical features extracted from data acquired by different scanners and protocols to improve stability and robustness. Previous studies have described various computational approaches to fuse single modality multicentre datasets. However, these surveys rarely focused on evaluation metrics and lacked a checklist for computational data harmonisation studies. In this systematic review, we summarise the computational data harmonisation approaches for multi-modality data in the digital healthcare field, including harmonisation strategies and evaluation metrics based on different theories. In addition, a comprehensive checklist that summarises common practices for data harmonisation studies is proposed to guide researchers to report their research findings more effectively. Last but not least, flowcharts presenting possible ways for methodology and metric selection are proposed and the limitations of different methods have been surveyed for future research.

Multiparametric Functional MRI of the Kidney: Current State and Future Trends with Deep Learning Approaches

BACKGROUND

 Until today, assessment of renal function has remained a challenge for modern medicine. In many cases, kidney diseases accompanied by a decrease in renal function remain undetected and unsolved, since neither laboratory tests nor imaging diagnostics provide adequate information on kidney status. In recent years, developments in the field of functional magnetic resonance imaging with application to abdominal organs have opened new possibilities combining anatomic imaging with multiparametric functional information. The multiparametric approach enables the measurement of perfusion, diffusion, oxygenation, and tissue characterization in one examination, thus providing more comprehensive insight into pathophysiological processes of diseases as well as effects of therapeutic interventions. However, application of multiparametric fMRI in the kidneys is still restricted mainly to research areas and transfer to the clinical routine is still outstanding. One of the major challenges is the lack of a standardized protocol for acquisition and postprocessing including efficient strategies for data analysis. This article provides an overview of the most common fMRI techniques with application to the kidney together with new approaches regarding data analysis with deep learning.

METHODS

 This article implies a selective literature review using the literature database PubMed in May 2021 supplemented by our own experiences in this field.

RESULTS AND CONCLUSION

 Functional multiparametric MRI is a promising technique for assessing renal function in a more comprehensive approach by combining multiple parameters such as perfusion, diffusion, and BOLD imaging. New approaches with the application of deep learning techniques could substantially contribute to overcoming the challenge of handling the quantity of data and developing more efficient data postprocessing and analysis protocols. Thus, it can be hoped that multiparametric fMRI protocols can be sufficiently optimized to be used for routine renal examination and to assist clinicians in the diagnostics, monitoring, and treatment of kidney diseases in the future.

KEY POINTS

  · Multiparametric fMRI is a technique performed without the use of radiation, contrast media, and invasive methods.. · Multiparametric fMRI provides more comprehensive insight into pathophysiological processes of kidney diseases by combining functional and structural parameters.. · For broader acceptance of fMRI biomarkers, there is a need for standardization of acquisition, postprocessing, and analysis protocols as well as more prospective studies.. · Deep learning techniques could significantly contribute to an optimization of data acquisition and the postprocessing and interpretation of larger quantities of data..

CITATION FORMAT

· Zhang C, Schwartz M, Küstner T et al. Multiparametric Functional MRI of the Kidney: Current State and Future Trends with Deep Learning Approaches. Fortschr Röntgenstr 2022; DOI: 10.1055/a-1775-8633.

Propagating Uncertainty Across Cascaded Medical Imaging Tasks for Improved Deep Learning Inference

Although deep networks have been shown to perform very well on a variety of medical imaging tasks, inference in the presence of pathology presents several challenges to common models. These challenges impede the integration of deep learning models into real clinical workflows, where the customary process of cascading deterministic outputs from a sequence of image-based inference steps (e.g. registration, segmentation) generally leads to an accumulation of errors that impacts the accuracy of downstream inference tasks. In this paper, we propose that by embedding uncertainty estimates across cascaded inference tasks, performance on the downstream inference tasks should be improved. We demonstrate the effectiveness of the proposed approach in three different clinical contexts: (i) We demonstrate that by propagating T2 weighted lesion segmentation results and their associated uncertainties, subsequent T2 lesion detection performance is improved when evaluated on a proprietary large-scale, multi-site, clinical trial dataset acquired from patients with Multiple Sclerosis. (ii) We show an improvement in brain tumour segmentation performance when the uncertainty map associated with a synthesised missing MR volume is provided as an additional input to a follow-up brain tumour segmentation network, when evaluated on the publicly available BraTS-2018 dataset. (iii) We show that by propagating uncertainties from a voxel-level hippocampus segmentation task, the subsequent regression of the Alzheimer's disease clinical score is improved.

Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis

Fusing multi-modality medical images, such as magnetic resonance (MR) imaging and positron emission tomography (PET), can provide various anatomical and functional information about the human body. However, PET data is not always available for several reasons, such as high cost, radiation hazard, and other limitations. This paper proposes a 3D end-to-end synthesis network called Bidirectional Mapping Generative Adversarial Networks (BMGAN). Image contexts and latent vectors are effectively used for brain MR-to-PET synthesis. Specifically, a bidirectional mapping mechanism is designed to embed the semantic information of PET images into the high-dimensional latent space. Moreover, the 3D Dense-UNet generator architecture and the hybrid loss functions are further constructed to improve the visual quality of cross-modality synthetic images. The most appealing part is that the proposed method can synthesize perceptually realistic PET images while preserving the diverse brain structures of different subjects. Experimental results demonstrate that the performance of the proposed method outperforms other competitive methods in terms of quantitative measures, qualitative displays, and evaluation metrics for classification.

Deep learning based MRI contrast synthesis using full volume prediction using full volume prediction

In Magnetic Resonance Imaging (MRI), depending on the image acquisition settings, a large number of image types or contrasts can be generated showing complementary information of the same imaged subject. This multi-spectral information is highly beneficial since can improve MRI analysis tasks such as segmentation and registration, thanks to pattern ambiguity reduction. However, the acquisition of several contrasts is not always possible due to time limitations and patient comfort constraints. Contrast synthesis has emerged recently as an approximate solution to generate other image types different from those acquired originally. Most of the previously proposed methods for contrast synthesis are slice-based which result in intensity inconsistencies between neighbor slices when applied in 3D. We propose the use of a 3D convolutional neural network (CNN) capable of generating T2 and FLAIR images from a single anatomical T1 source volume. The proposed network is a 3D variant of the UNet that processes the whole volume at once breaking with the inconsistency in the resulting output volumes related to 2D slice or patch-based methods. Since working with a full volume at once has a huge memory demand we have introduced a spatial-to-depth and a reconstruction layer that allows working with the full volume but maintain the required network complexity to solve the problem. Our approach enhances the coherence in the synthesized volume while improving the accuracy thanks to the integrated three-dimensional context-awareness. Finally, the proposed method has been validated with a segmentation method, thus demonstrating its usefulness in a direct and relevant application.

MRI classification using semantic random forest with auto-context model

BACKGROUND

It is challenging to differentiate air and bone on MR images of conventional sequences due to their low contrast. We propose to combine semantic feature extraction under auto-context manner into random forest to improve reasonability of the MRI segmentation for MRI-based radiotherapy treatment planning or PET attention correction.

METHODS

We applied a semantic classification random forest (SCRF) method which consists of a training stage and a segmentation stage. In the training stage, patch-based MRI features were extracted from registered MRI-CT training images, and the most informative elements were selected via feature selection to train an initial random forest. The rest sequence of random forests was trained by a combination of MRI feature and semantic feature under an auto-context manner. During segmentation, the MRI patches were first fed into these random forests to derive patch-based segmentation. By using patch fusion, the final end-to-end segmentation was obtained.

RESULTS

The Dice similarity coefficient (DSC) for air, bone and soft tissue classes obtained via proposed method were 0.976±0.007, 0.819±0.050 and 0.932±0.031, compared to 0.916±0.099, 0.673±0.151 and 0.830±0.083 with random forest (RF), and 0.942±0.086, 0.791±0.046 and 0.917±0.033 with U-Net. SCRF also outperformed the competing methods in sensitivity and specificity for all three structure types.

CONCLUSIONS

The proposed method accurately segmented bone, air and soft tissue. It is promising in facilitating advanced MR application in diagnosis and therapy.

Learning to Generate Missing Pulse Sequence in MRI using Deep Convolution Neural Network Trained with Visual Turing Test

Magnetic resonance imaging (MRI) is widely used in clinical applications due to its ability to acquire a wide variety of soft tissues using multiple pulse sequences. Each sequence provides information that generally complements the other. However, factors like an increase in scan time or contrast allergies impede imaging with numerous sequences. Synthesizing images of such non acquired sequences is a challenging proposition that can suffice for corrupted acquisition, fast reconstruction prior, super-resolution, etc. This manuscript employed a deep convolution neural network (CNN) to synthesize multiple missing pulse sequences of brain MRI with tumors. The CNN is an encoder-decoder-like network trained to minimize reconstruction mean square error (MSE) loss while maximizing the adversarial attack. It inflicts on a relativistic Visual Turing Test discriminator (rVTT). The approach is evaluated through experiments performed with the Brats2018 dataset, quantitative metrics viz. MSE, Structural Similarity Measure (SSIM), and Peak Signal to Noise Ratio (PSNR). The Radiologist and MR physicist performed the Turing test with 76% accuracy, demonstrating our approach's performance superiority over the prior art. We can synthesize MR images of missing pulse sequences at an inference cost of 350.71 GFlops/voxel through this approach.

How far spatial resolution affects the ensemble machine learning based flood susceptibility prediction in data sparse region

Although the effect of digital elevation model (DEM) and its spatial resolution on flood simulation modeling has been well studied, the effect of coarse and finer resolution image and DEM data on machine learning ensemble flood susceptibility prediction has not been investigated, particularly in data sparse conditions. The present work was, therefore, to investigate the performance of the resolution effects, such as coarse (Landsat and SRTM) and high (Sentinel-2 and ALOS PALSAR) resolution data on the flood susceptible models. Another motive of this study was to construct very high precision and robust flood susceptible models using standalone and ensemble machine learning algorithms. In the present study, fifteen flood conditioning parameters were generated from both coarse and high resolution datasets. Then, the ANN-multilayer perceptron (MLP), random forest (RF), bagging (B)-MLP, B-gaussian processes (B-GP) and B-SMOreg algorithms were used to integrate the flood conditioning parameters for generating the flood susceptible models. Furthermore, the influence of flood conditioning parameters on the modelling of flood susceptibility was investigated by proposing an ROC based sensitivity analysis. The validation of flood susceptibility models is also another challenge. In the present study, we proposed an index of flood vulnerability model to validate flood susceptibility models along with conventional statistical techniques, such as the ROC curve. Results showed that the coarse resolution based flood susceptibility MLP model has appeared as the best model (area under curve: 0.94) and it has predicted 11.65 % of the area as very high flood susceptible zones (FSz), followed by RF, B-MLP, B-GP, and B-SMOreg. Similarly, the high resolution based flood susceptibility model using MLP has predicted 19.34 % of areas as very high flood susceptible zones, followed by RF (14.32 %),B-MLP (14.88 %), B-GP, and B-SMOreg. On the other hand, ROC based sensitivity analysis showed that elevation influences flood susceptibility largely for coarse and high resolution based models, followed by drainage densityand flow accumulation. In addition, the accuracy assessment using the IFV model revealed that the MLP model outperformed all other models in the case of a high resolution imageThe coarser resolution image's performance level is acceptable but quite low. So, the study recommended the use of high resolution images for developing a machine learning algorithm based flood susceptibility model. As the study has clearly identified the areas of higher flood susceptibility and the dominant influencing factors for flooding, this could be used as a good database for flood management.

Anatomic and Molecular MR Image Synthesis Using Confidence Guided CNNs

Data-driven automatic approaches have demonstrated their great potential in resolving various clinical diagnostic dilemmas in neuro-oncology, especially with the help of standard anatomic and advanced molecular MR images. However, data quantity and quality remain a key determinant, and a significant limit of the potential applications. In our previous work, we explored the synthesis of anatomic and molecular MR image networks (SAMR) in patients with post-treatment malignant gliomas. In this work, we extend this through a confidence-guided SAMR (CG-SAMR) that synthesizes data from lesion contour information to multi-modal MR images, including T1-weighted ( [Formula: see text]), gadolinium enhanced [Formula: see text] (Gd- [Formula: see text]), T2-weighted ( [Formula: see text]), and fluid-attenuated inversion recovery ( FLAIR ), as well as the molecular amide proton transfer-weighted ( [Formula: see text]) sequence. We introduce a module that guides the synthesis based on a confidence measure of the intermediate results. Furthermore, we extend the proposed architecture to allow training using unpaired data. Extensive experiments on real clinical data demonstrate that the proposed model can perform better than current the state-of-the-art synthesis methods. Our code is available at https://github.com/guopengf/CG-SAMR.

Fat-saturated image generation from multi-contrast MRIs using generative adversarial networks with Bloch equation-based autoencoder regularization

Obtaining multiple series of magnetic resonance (MR) images with different contrasts is useful for accurate diagnosis of human spinal conditions. However, this can be time consuming and a burden on both the patient and the hospital. We propose a Bloch equation-based autoencoder regularization generative adversarial network (BlochGAN) to generate a fat saturation T2-weighted (T2 FS) image from T1-weighted (T1-w) and T2-weighted (T2-w) images of human spine. To achieve this, our approach was to utilize the relationship between the contrasts using Bloch equation since it is a fundamental principle of MR physics and serves as a physical basis of each contrasts. BlochGAN properly generated the target-contrast images using the autoencoder regularization based on the Bloch equation to identify the physical basis of the contrasts. BlochGAN consists of four sub-networks: an encoder, a decoder, a generator, and a discriminator. The encoder extracts features from the multi-contrast input images, and the generator creates target T2 FS images using the features extracted from the encoder. The discriminator assists network learning by providing adversarial loss, and the decoder reconstructs the input multi-contrast images and regularizes the learning process by providing reconstruction loss. The discriminator and the decoder are only used in the training process. Our results demonstrate that BlochGAN achieved quantitatively and qualitatively superior performance compared to conventional medical image synthesis methods in generating spine T2 FS images from T1-w, and T2-w images.

Machine learning algorithm-based risk assessment of riparian wetlands in Padma River Basin of Northwest Bangladesh

Wetland risk assessment is a global concern especially in developing countries like Bangladesh. The present study explored the spatiotemporal dynamics of wetlands, prediction of wetland risk assessment. The wetland risk assessment was predicted based on ten selected parameters, such as fragmentation probability, distance to road, and settlement. We used M5P, random forest (RF), reduced error pruning tree (REPTree), and support vector machine (SVM) machine learning techniques for wetland risk assessment. The results showed that wetland areas at present are declining less than one-third of those in 1988 due to the construction of the dam at Farakka, which is situated at the upstream of the Padma River. The distance to the river and built-up area are the two most contributing drivers influencing the wetland risk assessment based on information gain ratio (InGR). The prediction results of machine learning models showed 64.48% of area by M5P, 61.75% of area by RF, 62.18% of area by REPTree, and 55.74% of area by SVM have been predicted as the high and very high-risk zones. The results of accuracy assessment showed that the RF outperformed than other models (area under curve: 0.83), followed by the SVM, M5P, and REPTree. Degradation of wetlands explored in this study demonstrated the negative effects on biodiversity. Therefore, to conserve and protect the wetlands, continuous monitoring of wetlands using high resolution satellite images, feeding with the ecological flow, confining built up area and agricultural expansion towards wetlands, and new wetland creation is essential for wetland management.

Multi-modal MRI Image Synthesis via GAN with Multi-scale Gate Mergence

Multi-modal magnetic resonance imaging (MRI) plays a critical role in clinical diagnosis and treatment nowadays. Each modality of MRI presents its own specific anatomical features which serve as complementary information to other modalities and can provide rich diagnostic information. However, due to the limitations of time consuming and expensive cost, some image sequences of patients may be lost or corrupted, posing an obstacle for accurate diagnosis. Although current multi-modal image synthesis approaches are able to alleviate the issues to some extent, they are still far short of fusing modalities effectively. In light of this, we propose a multi-scale gate mergence based generative adversarial network model, namely MGM-GAN, to synthesize one modality of MRI from others. Notably, we have multiple down-sampling branches corresponding to input modalities to specifically extract their unique features. In contrast to the generic multi-modal fusion approach of averaging or maximizing operations, we introduce a gate mergence (GM) mechanism to automatically learn the weights of different modalities across locations, enhancing the task-related information while suppressing the irrelative information. As such, the feature maps of all the input modalities at each down-sampling level, i.e., multi-scale levels, are integrated via GM module. In addition, both the adversarial loss and the pixel-wise loss, as well as gradient difference loss (GDL) are applied to train the network to produce the desired modality accurately. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art multi-modal image synthesis methods.