Spatial-temporal modelling of fMRI data through spatially regularized mixture of hidden process models

DECT-CLUST: Dual-Energy CT Image Clustering and Application to Head and Neck Squamous Cell Carcinoma Segmentation

Dual-energy computed tomography (DECT) is an advanced CT computed tomography scanning technique enabling material characterization not possible with conventional CT scans. It allows the reconstruction of energy decay curves at each 3D image voxel, representing varied image attenuation at different effective scanning energy levels. In this paper, we develop novel unsupervised learning techniques based on mixture models and functional data analysis models to the clustering of DECT images. We design functional mixture models that integrate spatial image context in mixture weights, with mixture component densities being constructed upon the DECT energy decay curves as functional observations. We develop dedicated expectation-maximization algorithms for the maximum likelihood estimation of the model parameters. To our knowledge, this is the first article to develop statistical functional data analysis and model-based clustering techniques to take advantage of the full spectral information provided by DECT. We evaluate the application of DECT to head and neck squamous cell carcinoma. Current image-based evaluation of these tumors in clinical practice is largely qualitative, based on a visual assessment of tumor anatomic extent and basic one- or two-dimensional tumor size measurements. We evaluate our methods on 91 head and neck cancer DECT scans and compare our unsupervised clustering results to tumor contours traced manually by radiologists, as well as to several baseline algorithms. Given the inter-rater variability even among experts at delineating head and neck tumors, and given the potential importance of tissue reactions surrounding the tumor itself, our proposed methodology has the potential to add value in downstream machine learning applications for clinical outcome prediction based on DECT data in head and neck cancer.

Evolutional Neural Architecture Search for Optimization of Spatiotemporal Brain Network Decomposition

Using deep neural networks (DNNs) to explore spatial patterns and temporal dynamics of human brain activities has been an important yet challenging problem because the artificial neural networks are hard to be designed manually. There have been several promising deep learning methods, e.g., deep belief network (DBN), convolutional neural network (CNN), and deep sparse recurrent auto-encoder (DSRAE), that can decompose neuroscientific and meaningful spatiotemporal patterns from 4D functional Magnetic Resonance Imaging (fMRI) data. However, those previous studies still depend on hand-crafted neural network architectures and hyperparameters, which are not optimal in various senses. In this paper, we employ the evolutionary algorithms (EA) to optimize the deep neural architecture of DSRAE by minimizing the expected loss of initialized models, named eNAS-DSRAE (evolutionary Neural Architecture Search on Deep Sparse Recurrent Auto-Encoder). Also, validation experiments are designed and performed on the publicly available human connectome project (HCP) 900 datasets, and the results achieved by the optimized eNAS-DSRAE suggested that our framework can successfully identify the spatiotemporal features and perform better than the hand-crafted neural network models. To our best knowledge, the proposed eNAS-DSRAE is not only among the earliest NAS models that can extract connectome-scale meaningful spatiotemporal brain networks from 4D fMRI data, but also is an effective framework to optimize the RNN-based models.

Simultaneous spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder

Exploring the spatial patterns and temporal dynamics of human brain activity has been of great interest, in the quest to better understand connectome-scale brain networks. Though modeling spatial and temporal patterns of functional brain networks have been researched for a long time, the development of a unified and simultaneous spatial-temporal model has yet to be realized. For instance, although some deep learning methods have been proposed recently in order to model functional brain networks, most of them can only represent either spatial or temporal perspective of functional Magnetic Resonance Imaging (fMRI) data and rarely model both domains simultaneously. Due to the recent success in applying sequential auto-encoders for brain decoding, in this paper, we propose a deep sparse recurrent auto-encoder (DSRAE) to be applied unsupervised to learn spatial patterns and temporal fluctuations of brain networks at the same time. The proposed DSRAE was evaluated and validated based on three tasks of the publicly available Human Connectome Project (HCP) fMRI dataset, resulting with promising evidence. To the best of our knowledge, the proposed DSRAE is among the early efforts in developing unified models that can extract connectome-scale spatial-temporal networks from 4D fMRI data simultaneously.

4D Modeling of fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)

Since the human brain functional mechanism has been enabled for investigation by the functional Magnetic Resonance Imaging (fMRI) technology, simultaneous modeling of both the spatial and temporal patterns of brain functional networks from 4D fMRI data has been a fundamental but still challenging research topic for neuroimaging and medical image analysis fields. Currently, general linear model (GLM), independent component analysis (ICA), sparse dictionary learning, and recently deep learning models, are major methods for fMRI data analysis in either spatial or temporal domains, but there are few joint spatial-temporal methods proposed, as far as we know. As a result, the 4D nature of fMRI data has not been effectively investigated due to this methodological gap. The recent success of deep learning applications for functional brain decoding and encoding greatly inspired us in this work to propose a novel framework called spatio-temporal convolutional neural network (ST-CNN) to extract both spatial and temporal characteristics from targeted networks jointly and automatically identify of functional networks. The identification of Default Mode Network (DMN) from fMRI data was used for evaluation of the proposed framework. Results show that only training the framework on one fMRI dataset is sufficiently generalizable to identify the DMN from different datasets of different cognitive tasks and resting state. Further investigation of the results shows that the joint-learning scheme can capture the intrinsic relationship between the spatial and temporal characteristics of DMN and thus it ensures the accurate identification of DMN from independent datasets. The ST-CNN model brings new tools and insights for fMRI analysis in cognitive and clinical neuroscience studies.

Differentiable neural architecture search for optimal spatial/temporal brain function network decomposition

It has been a key topic to decompose the brain's spatial/temporal function networks from 4D functional magnetic resonance imaging (fMRI) data. With the advantages of robust and meaningful brain pattern extraction, deep neural networks have been shown to be more powerful and flexible in fMRI data modeling than other traditional methods. However, the challenge of designing neural network architecture for high-dimensional and complex fMRI data has also been realized recently. In this paper, we propose a new spatial/temporal differentiable neural architecture search algorithm (ST-DARTS) for optimal brain network decomposition. The core idea of ST-DARTS is to optimize the inner cell structure of the vanilla recurrent neural network (RNN) in order to effectively decompose spatial/temporal brain function networks from fMRI data. Based on the evaluations on all seven fMRI tasks in human connectome project (HCP) dataset, the ST-DARTS model is shown to perform promisingly, both spatially (i.e., it can recognize the most stimuli-correlated spatial brain network activation that is very similar to the benchmark) and temporally (i.e., its temporal activity is highly positively correlated with the task-design). To further improve the efficiency of ST-DARTS model, we introduce a flexible early-stopping mechanism, named as ST-DARTS+, which further improves experimental results significantly. To our best knowledge, the proposed ST-DARTS and ST-DARTS+ models are among the early efforts in optimally decomposing spatial/temporal brain function networks from fMRI data with neural architecture search strategy and they demonstrate great promise.

Structural impacts on the timing and amplitude of the negative BOLD response

The positive (PBR) and negative BOLD responses (NBR) arising in task-fMRI display varying magnitudes and dynamics across voxels. While the effects of structure, particularly of veins, on the PBR have been studied, little is known of NBR-structure relationships. Like the PBR, the NBR is often used as a surrogate marker of neuronal activation in both basic and clinical research and assessing its relationship with cortical structure may help interpret group differences. We therefore investigated how local structure affects BOLD amplitude and timing in PBR and NBR areas using multi-band fMRI during visual stimulation to obtain high temporal resolution (TR=0.45s) data combined with T1 imaging and susceptibility-weighted imaging (SWI) to quantify the local densities of gray/white matter and veins, respectively. In both PBRs and NBRs, larger venous density was consistently associated with larger BOLD amplitude and delay, up to 1-2s larger relative to areas devoid of large veins. Both binary and sinusoidal visual stimulus modulation yielded similar activation maps and results, suggesting that underlying vasculature affects PBR and NBR temporal dynamics in the same manner. Accounting for structural impacts on PBR and NBR magnitude and timing could help enhance activation map accuracy, better assess functional connectivity, and better characterize neurovascular coupling.

Statistical modeling of time-dependent fMRI activation effects

Functional magnetic resonance imaging (fMRI) activation detection within stimulus-based experimental paradigms is conventionally based on the assumption that activation effects remain constant over time. This assumption neglects the fact that the strength of activation may vary, for example, due to habituation processes or changing attention. Neither the functional form of time variation can be retrieved nor short-lasting effects can be detected by conventional methods. In this work, a new dynamic approach is proposed that allows to estimate time-varying effect profiles and hemodynamic response functions in event-related fMRI paradigms. To this end, we incorporate the time-varying coefficient methodology into the fMRI general regression framework. Inference is based on a voxelwise penalized least squares procedure. We assess the strength of activation and corresponding time variation on the basis of pointwise confidence intervals on a voxel level. Additionally, spatial clusters of effect curves are presented. Results of the analysis of an active oddball experiment show that activation effects deviating from a constant trend coexist with time-varying effects that exhibit different types of shapes, such as linear, (inversely) U-shaped or fluctuating forms. In a comparison to conventional approaches, like classical SPM, we observe that time-constant methods are rather insensitive to detect temporary effects, because these do not emerge when aggregated across the entire experiment. Hence, it is recommended to base activation detection analyses not merely on time-constant procedures but to include flexible time-varying effects that harbour valuable information on individual response patterns.