1
|
Ling Q, Liu A, Li Y, McKeown MJ, Chen X. fMRI-based spatio-temporal parcellations of the human brain. Curr Opin Neurol 2024; 37:369-380. [PMID: 38804205 DOI: 10.1097/wco.0000000000001280] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
PURPOSE OF REVIEW Human brain parcellation based on functional magnetic resonance imaging (fMRI) plays an essential role in neuroscience research. By segmenting vast and intricate fMRI data into functionally similar units, researchers can better decipher the brain's structure in both healthy and diseased states. This article reviews current methodologies and ideas in this field, while also outlining the obstacles and directions for future research. RECENT FINDINGS Traditional brain parcellation techniques, which often rely on cytoarchitectonic criteria, overlook the functional and temporal information accessible through fMRI. The adoption of machine learning techniques, notably deep learning, offers the potential to harness both spatial and temporal information for more nuanced brain segmentation. However, the search for a one-size-fits-all solution to brain segmentation is impractical, with the choice between group-level or individual-level models and the intended downstream analysis influencing the optimal parcellation strategy. Additionally, evaluating these models is complicated by our incomplete understanding of brain function and the absence of a definitive "ground truth". SUMMARY While recent methodological advancements have significantly enhanced our grasp of the brain's spatial and temporal dynamics, challenges persist in advancing fMRI-based spatio-temporal representations. Future efforts will likely focus on refining model evaluation and selection as well as developing methods that offer clear interpretability for clinical usage, thereby facilitating further breakthroughs in our comprehension of the brain.
Collapse
Affiliation(s)
- Qinrui Ling
- Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230027, China
| | - Aiping Liu
- Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230027, China
| | - Yu Li
- Institute of Dataspace, Hefei Comprehensive National Science Center, Hefei 230088, China
| | - Martin J McKeown
- Department of Medicine, University of British Columbia, Vancouver, Vancouver V6T2B5, Canada
| | - Xun Chen
- Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230027, China
| |
Collapse
|
2
|
Zhu X, Kim Y, Ravid O, He X, Suarez-Jimenez B, Zilcha-Mano S, Lazarov A, Lee S, Abdallah CG, Angstadt M, Averill CL, Baird CL, Baugh LA, Blackford JU, Bomyea J, Bruce SE, Bryant RA, Cao Z, Choi K, Cisler J, Cotton AS, Daniels JK, Davenport ND, Davidson RJ, DeBellis MD, Dennis EL, Densmore M, deRoon-Cassini T, Disner SG, Hage WE, Etkin A, Fani N, Fercho KA, Fitzgerald J, Forster GL, Frijling JL, Geuze E, Gonenc A, Gordon EM, Gruber S, Grupe DW, Guenette JP, Haswell CC, Herringa RJ, Herzog J, Hofmann DB, Hosseini B, Hudson AR, Huggins AA, Ipser JC, Jahanshad N, Jia-Richards M, Jovanovic T, Kaufman ML, Kennis M, King A, Kinzel P, Koch SBJ, Koerte IK, Koopowitz SM, Korgaonkar MS, Krystal JH, Lanius R, Larson CL, Lebois LAM, Li G, Liberzon I, Lu GM, Luo Y, Magnotta VA, Manthey A, Maron-Katz A, May G, McLaughlin K, Mueller SC, Nawijn L, Nelson SM, Neufeld RWJ, Nitschke JB, O'Leary EM, Olatunji BO, Olff M, Peverill M, Phan KL, Qi R, Quidé Y, Rektor I, Ressler K, Riha P, Ross M, Rosso IM, Salminen LE, Sambrook K, Schmahl C, Shenton ME, Sheridan M, Shih C, Sicorello M, Sierk A, Simmons AN, Simons RM, Simons JS, Sponheim SR, Stein MB, Stein DJ, Stevens JS, Straube T, Sun D, Théberge J, Thompson PM, Thomopoulos SI, van der Wee NJA, van der Werff SJA, van Erp TGM, van Rooij SJH, van Zuiden M, Varkevisser T, Veltman DJ, Vermeiren RRJM, Walter H, Wang L, Wang X, Weis C, Winternitz S, Xie H, Zhu Y, Wall M, Neria Y, Morey RA. Neuroimaging-based classification of PTSD using data-driven computational approaches: A multisite big data study from the ENIGMA-PGC PTSD consortium. Neuroimage 2023; 283:120412. [PMID: 37858907 PMCID: PMC10842116 DOI: 10.1016/j.neuroimage.2023.120412] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 09/10/2023] [Accepted: 10/16/2023] [Indexed: 10/21/2023] Open
Abstract
BACKGROUND Recent advances in data-driven computational approaches have been helpful in devising tools to objectively diagnose psychiatric disorders. However, current machine learning studies limited to small homogeneous samples, different methodologies, and different imaging collection protocols, limit the ability to directly compare and generalize their results. Here we aimed to classify individuals with PTSD versus controls and assess the generalizability using a large heterogeneous brain datasets from the ENIGMA-PGC PTSD Working group. METHODS We analyzed brain MRI data from 3,477 structural-MRI; 2,495 resting state-fMRI; and 1,952 diffusion-MRI. First, we identified the brain features that best distinguish individuals with PTSD from controls using traditional machine learning methods. Second, we assessed the utility of the denoising variational autoencoder (DVAE) and evaluated its classification performance. Third, we assessed the generalizability and reproducibility of both models using leave-one-site-out cross-validation procedure for each modality. RESULTS We found lower performance in classifying PTSD vs. controls with data from over 20 sites (60 % test AUC for s-MRI, 59 % for rs-fMRI and 56 % for d-MRI), as compared to other studies run on single-site data. The performance increased when classifying PTSD from HC without trauma history in each modality (75 % AUC). The classification performance remained intact when applying the DVAE framework, which reduced the number of features. Finally, we found that the DVAE framework achieved better generalization to unseen datasets compared with the traditional machine learning frameworks, albeit performance was slightly above chance. CONCLUSION These results have the potential to provide a baseline classification performance for PTSD when using large scale neuroimaging datasets. Our findings show that the control group used can heavily affect classification performance. The DVAE framework provided better generalizability for the multi-site data. This may be more significant in clinical practice since the neuroimaging-based diagnostic DVAE classification models are much less site-specific, rendering them more generalizable.
Collapse
Affiliation(s)
- Xi Zhu
- Department of Psychiatry, Columbia University Medical Center, New York, NY, USA; New York State Psychiatric Institute, New York, NY, USA
| | - Yoojean Kim
- New York State Psychiatric Institute, New York, NY, USA
| | - Orren Ravid
- New York State Psychiatric Institute, New York, NY, USA
| | - Xiaofu He
- Department of Psychiatry, Columbia University Medical Center, New York, NY, USA
| | | | | | | | - Seonjoo Lee
- Department of Psychiatry, Columbia University Medical Center, New York, NY, USA; New York State Psychiatric Institute, New York, NY, USA
| | - Chadi G Abdallah
- Baylor College of Medicine, Houston, TX, USA; Yale University School of Medicine, New Haven, CT, USA
| | | | - Christopher L Averill
- Baylor College of Medicine, Houston, TX, USA; Yale University School of Medicine, New Haven, CT, USA
| | | | - Lee A Baugh
- Sanford School of Medicine, University of South Dakota, Vermillion, SD, USA
| | | | | | - Steven E Bruce
- Center for Trauma Recovery, Department of Psychological Sciences, University of Missouri-St. Louis, St. Louis, MO, USA
| | - Richard A Bryant
- School of Psychology, University of New South Wales, Sydney, NSW, Australia
| | - Zhihong Cao
- Department of Radiology, The Affiliated Yixing Hospital of Jiangsu University, Yixing, Jiangsu, China
| | - Kyle Choi
- University of California San Diego, La Jolla, CA, USA
| | - Josh Cisler
- Department of Psychiatry, University of Texas at Austin, Austin, TX, USA
| | | | | | | | | | | | - Emily L Dennis
- University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Maria Densmore
- Departments of Psychology and Psychiatry, Neuroscience Program, Western University, London, ON, Canada; Department of Psychology, University of British Columbia, Okanagan, Kelowna, British Columbia, Canada
| | | | - Seth G Disner
- Minneapolis VA Health Care System, Minneapolis, MN, USA
| | - Wissam El Hage
- UMR 1253, CIC 1415, University of Tours, CHRU de Tours, INSERM, France
| | | | - Negar Fani
- Emory University Department of Psychiatry and Behavioral Sciences, Atlanta, GA, USA
| | - Kelene A Fercho
- Civil Aerospace Medical Institute, US Federal Aviation Administration, Oklahoma City, OK, USA
| | | | - Gina L Forster
- Brain Health Research Centre, Department of Anatomy, University of Otago, Dunedin, New Zealand
| | - Jessie L Frijling
- Department of Psychiatry, Amsterdam University Medical Centers, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
| | - Elbert Geuze
- Brain Research and Innovation Centre, Ministry of Defence, Utrecht, The Netherlands
| | - Atilla Gonenc
- Cognitive and Clinical Neuroimaging Core, McLean Hospital, Belmont, MA, USA
| | - Evan M Gordon
- Department of Radiology, Washington University School of Medicine, St. Louis, MO, USA
| | - Staci Gruber
- Cognitive and Clinical Neuroimaging Core, McLean Hospital, Belmont, MA, USA
| | | | - Jeffrey P Guenette
- Division of Neuroradiology, Brigham and Women's Hospital, Boston, MA, USA
| | | | - Ryan J Herringa
- School of Medicine and Public Health, University of Wisconsin-Madison, Madison, WI, USA
| | | | | | | | | | | | | | - Neda Jahanshad
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of the University of Southern California, Marina del Rey, CA, USA
| | | | | | - Milissa L Kaufman
- Division of Women's Mental Health, McLean Hospital, Belmont, MA, USA
| | - Mitzy Kennis
- Brain Research and Innovation Centre, Ministry of Defence, Utrecht, The Netherlands
| | | | - Philipp Kinzel
- Department of Child and Adolescent Psychiatry, Psychosomatic and Psychotherapy, Ludwig Maximilian University of Munich, Munich, Germany; Psychiatry Neuroimaging Laboratory, Brigham and Women's Hospital, Boston, MA, USA
| | - Saskia B J Koch
- Donders Institute for Brain, Cognition and Behavior, Centre for Cognitive Neuroimaging, Radboud University Nijmegen, Nijmegen, The Netherlands
| | - Inga K Koerte
- Department of Child and Adolescent Psychiatry, Psychosomatic and Psychotherapy, Ludwig Maximilian University of Munich, Munich, Germany; Psychiatry Neuroimaging Laboratory, Brigham and Women's Hospital, Boston, MA, USA
| | | | | | | | - Ruth Lanius
- Department of Neuroscience, Western University, London, ON, Canada
| | | | - Lauren A M Lebois
- McLean Hospital, Belmont, MA, USA; Harvard Medical School, Boston, MA, USA
| | - Gen Li
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
| | - Israel Liberzon
- Psychiatry and Behavioral Science, Texas A&M University Health Science Center, College Station, TX, USA
| | - Guang Ming Lu
- Department of Medical Imaging, Jinling Hospital, Medical School of Nanjing University, Nanjing, Jiangsu, China
| | - Yifeng Luo
- Department of Radiology, The Affiliated Yixing Hospital of Jiangsu University, Yixing, Jiangsu, China
| | | | - Antje Manthey
- Charité Universitätsmedizin Berlin Campus Charite Mitte: Charite Universitatsmedizin Berlin, Berlin, Germany
| | | | - Geoffery May
- VISN 17 Center of Excellence for Research on Returning War Veterans, Waco, TX, USA
| | | | | | - Laura Nawijn
- Department of Psychiatry, Amsterdam University Medical Centers, VU University Medical Center, VU University, Amsterdam, The Netherlands
| | - Steven M Nelson
- Department of Pediatrics, University of Minnesota, Minneapolis, MN, USA
| | - Richard W J Neufeld
- Departments of Psychology and Psychiatry, Neuroscience Program, Western University, London, ON, Canada; Department of Psychology, University of British Columbia, Okanagan, Kelowna, British Columbia, Canada
| | | | | | - Bunmi O Olatunji
- Department of Psychology, Vanderbilt University, Nashville, TN, USA
| | - Miranda Olff
- Department of Psychiatry, Amsterdam University Medical Centers, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
| | | | - K Luan Phan
- Department of Psychiatry and Behavioral Health, Ohio State University, Columbus, OH, USA
| | - Rongfeng Qi
- Department of Medical Imaging, Jinling Hospital, Medical School of Nanjing University, Nanjing, Jiangsu, China
| | - Yann Quidé
- School of Psychology, University of New South Wales, Sydney, NSW, Australia; Neuroscience Research Australia, Randwick, NSW, Australia
| | | | - Kerry Ressler
- McLean Hospital, Belmont, MA, USA; Harvard Medical School, Boston, MA, USA
| | | | - Marisa Ross
- Northwestern Neighborhood and Networks Initiative, Northwestern University Institute for Policy Research, Evanston, IL, USA
| | - Isabelle M Rosso
- McLean Hospital, Belmont, MA, USA; Harvard Medical School, Boston, MA, USA
| | - Lauren E Salminen
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of the University of Southern California, Marina del Rey, CA, USA
| | | | | | - Martha E Shenton
- Psychiatry Neuroimaging Laboratory, Brigham and Women's Hospital, Boston, MA, USA
| | | | | | | | - Anika Sierk
- Charité Universitätsmedizin Berlin Campus Charite Mitte: Charite Universitatsmedizin Berlin, Berlin, Germany
| | - Alan N Simmons
- Center of Excellence for Stress and Mental Health, VA San Diego Healthcare System, San Diego, CA, USA
| | | | | | - Scott R Sponheim
- Minneapolis VA Health Care System, Minneapolis, MN, USA; University of Minnesota, Minneapolis, MN, USA
| | | | - Dan J Stein
- University of Cape Town, Cape Town, South Africa
| | - Jennifer S Stevens
- Emory University Department of Psychiatry and Behavioral Sciences, Atlanta, GA, USA
| | | | | | - Jean Théberge
- Departments of Psychology and Psychiatry, Neuroscience Program, Western University, London, ON, Canada; Department of Psychology, University of British Columbia, Okanagan, Kelowna, British Columbia, Canada
| | - Paul M Thompson
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of the University of Southern California, Marina del Rey, CA, USA
| | - Sophia I Thomopoulos
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of the University of Southern California, Marina del Rey, CA, USA
| | | | | | | | - Sanne J H van Rooij
- Emory University Department of Psychiatry and Behavioral Sciences, Atlanta, GA, USA
| | - Mirjam van Zuiden
- Department of Psychiatry, Amsterdam University Medical Centers, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
| | - Tim Varkevisser
- Brain Research and Innovation Centre, Ministry of Defence, Utrecht, The Netherlands
| | - Dick J Veltman
- Department of Psychiatry, Amsterdam University Medical Centers, VU University Medical Center, VU University, Amsterdam, The Netherlands
| | | | - Henrik Walter
- Charité Universitätsmedizin Berlin Campus Charite Mitte: Charite Universitatsmedizin Berlin, Berlin, Germany
| | - Li Wang
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Xin Wang
- University of Toledo, Toledo, OH, USA
| | - Carissa Weis
- Medical College of Wisconsin, Milwaukee, WI, USA
| | - Sherry Winternitz
- Division of Women's Mental Health, McLean Hospital, Belmont, MA, USA
| | - Hong Xie
- University of Toledo, Toledo, OH, USA
| | - Ye Zhu
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
| | - Melanie Wall
- Department of Psychiatry, Columbia University Medical Center, New York, NY, USA; New York State Psychiatric Institute, New York, NY, USA
| | - Yuval Neria
- Department of Psychiatry, Columbia University Medical Center, New York, NY, USA
| | | |
Collapse
|
3
|
Yuan C, Duan J, Tustison NJ, Xu K, Hubbard RA, Linn KA. ReMiND: Recovery of Missing Neuroimaging using Diffusion Models with Application to Alzheimer's Disease. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.08.16.23294169. [PMID: 37662259 PMCID: PMC10473806 DOI: 10.1101/2023.08.16.23294169] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
Objective Missing data is a significant challenge in medical research. In longitudinal studies of Alzheimer's disease (AD) where structural magnetic resonance imaging (MRI) is collected from individuals at multiple time points, participants may miss a study visit or drop out. Additionally, technical issues such as participant motion in the scanner may result in unusable imaging data at designated visits. Such missing data may hinder the development of high-quality imaging-based biomarkers. Furthermore, when imaging data are unavailable in clinical practice, patients may not benefit from effective application of biomarkers for disease diagnosis and monitoring. Methods To address the problem of missing MRI data in studies of AD, we introduced a novel 3D diffusion model specifically designed for imputing missing structural MRI (Recovery of Missing Neuroimaging using Diffusion models (ReMiND)). The model generates a whole-brain image conditional on a single structural MRI observed at a past visit or conditional on one past and one future observed structural MRI relative to the missing observation. Results Experimental results show that our method can generate high-quality individual 3D structural MRI with high similarity to ground truth, observed images. Additionally, images generated using ReMiND exhibit relatively lower error rates and more accurately estimated rates of atrophy over time in important anatomical brain regions compared with two alternative imputation approaches: forward filling and image generation using variational autoencoders. Conclusion Our 3D diffusion model can impute missing structural MRI data at a single designated visit and outperforms alternative methods for imputing whole-brain images that are missing from longitudinal trajectories.
Collapse
Affiliation(s)
- Chenxi Yuan
- Department of Biostatistics, Epidemiology & Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, 19104, PA, USA
- Penn Statistics in Imaging and Visualization Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, 19104, PA, USA
| | - Jinhao Duan
- Department of Computer Science, College of Computing & Informatics, Drexel University, Philadelphia, USA, Philadelphia, 19104, PA, USA
| | - Nicholas J. Tustison
- Department of Radiology and Medical Imaging, School of Medicine, University of Virginia, Charlottesville, 22908, VA, USA
| | - Kaidi Xu
- Department of Computer Science, College of Computing & Informatics, Drexel University, Philadelphia, USA, Philadelphia, 19104, PA, USA
| | - Rebecca A. Hubbard
- Department of Biostatistics, Epidemiology & Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, 19104, PA, USA
| | - Kristin A. Linn
- Department of Biostatistics, Epidemiology & Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, 19104, PA, USA
- Penn Statistics in Imaging and Visualization Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, 19104, PA, USA
| |
Collapse
|
4
|
Jiang X, Yan J, Zhao Y, Jiang M, Chen Y, Zhou J, Xiao Z, Wang Z, Zhang R, Becker B, Zhu D, Kendrick KM, Liu T. Characterizing functional brain networks via Spatio-Temporal Attention 4D Convolutional Neural Networks (STA-4DCNNs). Neural Netw 2023; 158:99-110. [PMID: 36446159 DOI: 10.1016/j.neunet.2022.11.004] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Revised: 08/17/2022] [Accepted: 11/03/2022] [Indexed: 11/11/2022]
Abstract
Characterizing individualized spatio-temporal patterns of functional brain networks (FBNs) via functional magnetic resonance imaging (fMRI) provides a foundation for understanding complex brain function. Although previous studies have achieved promising performances based on either shallow or deep learning models, there is still much space to improve the accuracy of spatio-temporal pattern characterization of FBNs by optimally integrating the four-dimensional (4D) features of fMRI. In this study, we introduce a novel Spatio-Temporal Attention 4D Convolutional Neural Network (STA-4DCNN) model to characterize individualized spatio-temporal patterns of FBNs. Particularly, STA-4DCNN is composed of two subnetworks, in which the first Spatial Attention 4D CNN (SA-4DCNN) models the spatio-temporal features of 4D fMRI data and then characterizes the spatial pattern of FBNs, and the second Temporal Guided Attention Network (T-GANet) further characterizes the temporal pattern of FBNs under the guidance of the spatial pattern together with 4D fMRI data. We evaluate the proposed STA-4DCNN on seven different task fMRI and one resting state fMRI datasets from the publicly released Human Connectome Project. The experimental results demonstrate that STA-4DCNN has superior ability and generalizability in characterizing individualized spatio-temporal patterns of FBNs when compared to other state-of-the-art models. We further apply STA-4DCNN on another independent ABIDE I resting state fMRI dataset including both autism spectrum disorder (ASD) and typical developing (TD) subjects, and successfully identify abnormal spatio-temporal patterns of FBNs in ASD compared to TD. In general, STA-4DCNN provides a powerful tool for FBN characterization and for clinical applications on brain disease characterization at the individual level.
Collapse
Affiliation(s)
- Xi Jiang
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Jiadong Yan
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Yu Zhao
- Syngo Innovation, Siemens Healthineers, Malvern, PA 19355, USA
| | - Mingxin Jiang
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Yuzhong Chen
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Jingchao Zhou
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Zhenxiang Xiao
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Zifan Wang
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Rong Zhang
- Neuroscience Research Institute, Key Laboratory for Neuroscience, Ministry of Education of China, China; Key Laboratory for Neuroscience, National Committee of Health and Family Planning of China, China; Department of Neurobiology, School of Basic Medical Sciences, Peking University, Beijing, China
| | - Benjamin Becker
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Dajiang Zhu
- Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX 76019, USA
| | - Keith M Kendrick
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China.
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, USA.
| |
Collapse
|
5
|
Tu L, Talbot A, Gallagher NM, Carlson DE. Supervising the Decoder of Variational Autoencoders to Improve Scientific Utility. IEEE TRANSACTIONS ON SIGNAL PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2022; 70:5954-5966. [PMID: 36777018 PMCID: PMC9910304 DOI: 10.1109/tsp.2022.3230329] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Probabilistic generative models are attractive for scientific modeling because their inferred parameters can be used to generate hypotheses and design experiments. This requires that the learned model provides an accurate representation of the input data and yields a latent space that effectively predicts outcomes relevant to the scientific question. Supervised Variational Autoencoders (SVAEs) have previously been used for this purpose, as a carefully designed decoder can be used as an interpretable generative model of the data, while the supervised objective ensures a predictive latent representation. Unfortunately, the supervised objective forces the encoder to learn a biased approximation to the generative posterior distribution, which renders the generative parameters unreliable when used in scientific models. This issue has remained undetected as reconstruction losses commonly used to evaluate model performance do not detect bias in the encoder. We address this previously-unreported issue by developing a second-order supervision framework (SOS-VAE) that updates the decoder parameters, rather than the encoder, to induce a predictive latent representation. This ensures that the encoder maintains a reliable posterior approximation and the decoder parameters can be effectively interpreted. We extend this technique to allow the user to trade-off the bias in the generative parameters for improved predictive performance, acting as an intermediate option between SVAEs and our new SOS-VAE. We also use this methodology to address missing data issues that often arise when combining recordings from multiple scientific experiments. We demonstrate the effectiveness of these developments using synthetic data and electrophysiological recordings with an emphasis on how our learned representations can be used to design scientific experiments.
Collapse
Affiliation(s)
- Liyun Tu
- School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, China
| | - Austin Talbot
- Department of Psychiatry and Behavioral Sciences, Stanford University, Palo Alto, CA 94305, USA
| | - Neil M. Gallagher
- Department of Psychiatry, Weill Cornell Medical College, New York, NY 10065, USA
| | - David E. Carlson
- Department of Biostatistics and Bioinformatics and the Department of Civil and Environmental Engineering, Duke University, Durham, NC 27708, USA
| |
Collapse
|
6
|
Qiang N, Dong Q, Liang H, Li J, Zhang S, Zhang C, Ge B, Sun Y, Gao J, Liu T, Yue H, Zhao S. Learning brain representation using recurrent Wasserstein generative adversarial net. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 223:106979. [PMID: 35792364 DOI: 10.1016/j.cmpb.2022.106979] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 04/26/2022] [Accepted: 06/26/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE To understand brain cognition and disorders, modeling the mapping between mind and brain has been of great interest to the neuroscience community. The key is the brain representation, including functional brain networks (FBN) and their corresponding temporal features. Recently, it has been proven that deep learning models have superb representation power on functional magnetic resonance imaging (fMRI) over traditional machine learning methods. However, due to the lack of high-quality data and labels, deep learning models tend to suffer from overfitting in the training process. METHODS In this work, we applied a recurrent Wasserstein generative adversarial net (RWGAN) to learn brain representation from volumetric fMRI data. Generative adversarial net (GAN) is widely used in natural image generation and is able to capture the distribution of the input data, which enables the extraction of generalized features from fMRI and thus relieves the overfitting issue. The recurrent layers in RWGAN are designed to better model the local temporal features of the fMRI time series. The discriminator of RWGAN works as a deep feature extractor. With LASSO regression, the RWGAN model can decompose the fMRI data into temporal features and spatial features (FBNs). Furthermore, the generator of RWGAN can generate high-quality new data for fMRI augmentation. RESULTS The experimental results on seven tasks from the HCP dataset showed that the RWGAN can learn meaningful and interpretable temporal features and FBNs, compared to HCP task designs and general linear model (GLM) derived networks. Besides, the results on different training datasets showed that the RWGAN performed better on small datasets than other deep learning models. Moreover, we used the generator of RWGAN to yield fake subjects. The result showed that the fake data can also be used to learn meaningful representation compared to those learned from real data. CONCLUSIONS To our best knowledge, this work is among the earliest attempts of applying generative deep learning for modeling fMRI data. The proposed RWGAN offers a novel methodology for learning brain representation from fMRI, and it can generate high-quality fake data for the potential use of fMRI data augmentation.
Collapse
Affiliation(s)
- Ning Qiang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China; Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, China
| | - Qinglin Dong
- Advanced Medical Computing and Analysis, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | - Hongtao Liang
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Jin Li
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Shu Zhang
- Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, China
| | - Cheng Zhang
- School of Electronics Engineering and Computer Science, Peking University, Beijing, China
| | - Bao Ge
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China; Center for Brain and Brain-Inspired Computing Research, Department of Computer Science, Northwestern Polytechnical University, Xi'an, China
| | - Yifei Sun
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Jie Gao
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, Greece
| | - Huiji Yue
- School of Physics and Information Technology, Shaanxi Normal University, Xi'an, China.
| | - Shijie Zhao
- School of Automation, Northwestern Polytechnical University, Xi'an, China.
| |
Collapse
|
7
|
Panda R, Kalmady SV, Greiner R. Multi-Source Domain Adaptation Techniques for Mitigating Batch Effects: A Comparative Study. Front Neuroinform 2022; 16:805117. [PMID: 35528213 PMCID: PMC9067602 DOI: 10.3389/fninf.2022.805117] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Accepted: 03/15/2022] [Indexed: 11/13/2022] Open
Abstract
The past decade has seen an increasing number of applications of deep learning (DL) techniques to biomedical fields, especially in neuroimaging-based analysis. Such DL-based methods are generally data-intensive and require a large number of training instances, which might be infeasible to acquire from a single acquisition site, especially for data, such as fMRI scans, due to the time and costs that they demand. We can attempt to address this issue by combining fMRI data from various sites, thereby creating a bigger heterogeneous dataset. Unfortunately, the inherent differences in the combined data, known as batch effects, often hamper learning a model. To mitigate this issue, techniques such as multi-source domain adaptation [Multi-source Domain Adversarial Networks (MSDA)] aim at learning an effective classification function that uses (learned) domain-invariant latent features. This article analyzes and compares the performance of various popular MSDA methods [MDAN, Domain AggRegation Networks (DARN), Multi-Domain Matching Networks (MDMN), and Moment Matching for MSDA (M3SDA)] at predicting different labels (illness, age, and sex) of images from two public rs-fMRI datasets: ABIDE 1and ADHD-200. It also evaluates the impact of various conditions such as class imbalance, the number of sites along with a comparison of the degree of adaptation of each of the methods, thereby presenting the effectiveness of MSDA models in neuroimaging-based applications.
Collapse
Affiliation(s)
- Rohan Panda
- Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Sunil Vasu Kalmady
- Canadian VIGOUR Centre, University of Alberta, Edmonton, AB, Canada
- Department of Computing Science, University of Alberta, Edmonton, AB, Canada
| | - Russell Greiner
- Department of Computing Science, University of Alberta, Edmonton, AB, Canada
- Alberta Machine Intelligence Institute, Edmonton, AB, Canada
- Department of Psychiatry, University of Alberta, Edmonton, AB, Canada
| |
Collapse
|
8
|
Zhao Y, Li X, Huang H, Zhang W, Zhao S, Makkie M, Zhang M, Li Q, Liu T. 4D Modeling of fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN). IEEE Trans Cogn Dev Syst 2021; 12:451-460. [PMID: 33748420 DOI: 10.1109/tcds.2019.2916916] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
Since the human brain functional mechanism has been enabled for investigation by the functional Magnetic Resonance Imaging (fMRI) technology, simultaneous modeling of both the spatial and temporal patterns of brain functional networks from 4D fMRI data has been a fundamental but still challenging research topic for neuroimaging and medical image analysis fields. Currently, general linear model (GLM), independent component analysis (ICA), sparse dictionary learning, and recently deep learning models, are major methods for fMRI data analysis in either spatial or temporal domains, but there are few joint spatial-temporal methods proposed, as far as we know. As a result, the 4D nature of fMRI data has not been effectively investigated due to this methodological gap. The recent success of deep learning applications for functional brain decoding and encoding greatly inspired us in this work to propose a novel framework called spatio-temporal convolutional neural network (ST-CNN) to extract both spatial and temporal characteristics from targeted networks jointly and automatically identify of functional networks. The identification of Default Mode Network (DMN) from fMRI data was used for evaluation of the proposed framework. Results show that only training the framework on one fMRI dataset is sufficiently generalizable to identify the DMN from different datasets of different cognitive tasks and resting state. Further investigation of the results shows that the joint-learning scheme can capture the intrinsic relationship between the spatial and temporal characteristics of DMN and thus it ensures the accurate identification of DMN from independent datasets. The ST-CNN model brings new tools and insights for fMRI analysis in cognitive and clinical neuroscience studies.
Collapse
Affiliation(s)
- Yu Zhao
- Cortical Architecture Imaging and Discovery (CAID) Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| | - Xiang Li
- Massachusetts General Hospital and Harvard Medical School, Boston MA 02115, USA
| | - Heng Huang
- School of Automation, Northwestern Polytechnical University, Xi'an, Sha'anxi 710072, China
| | - Wei Zhang
- Cortical Architecture Imaging and Discovery (CAID) Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| | - Shijie Zhao
- School of Automation, Northwestern Polytechnical University, Xi'an, Sha'anxi 710072, China
| | - Milad Makkie
- Cortical Architecture Imaging and Discovery (CAID) Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| | - Mo Zhang
- Center for Data Science, Peking University, Beijing 100080, China
| | - Quanzheng Li
- Massachusetts General Hospital and Harvard Medical School, Boston MA 02115, USA.; Peking University, Laboratory for Biomedical Image Analysis, Beijing Institute of Big Data Research, Beijing 100871, China
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| |
Collapse
|
9
|
Abstract
Human brain atlases have been evolving tremendously, propelled recently by brain big projects, and driven by sophisticated imaging techniques, advanced brain mapping methods, vast data, analytical strategies, and powerful computing. We overview here this evolution in four categories: content, applications, functionality, and availability, in contrast to other works limited mostly to content. Four atlas generations are distinguished: early cortical maps, print stereotactic atlases, early digital atlases, and advanced brain atlas platforms, and 5 avenues in electronic atlases spanning the last two generations. Content-wise, new electronic atlases are categorized into eight groups considering their scope, parcellation, modality, plurality, scale, ethnicity, abnormality, and a mixture of them. Atlas content developments in these groups are heading in 23 various directions. Application-wise, we overview atlases in neuroeducation, research, and clinics, including stereotactic and functional neurosurgery, neuroradiology, neurology, and stroke. Functionality-wise, tools and functionalities are addressed for atlas creation, navigation, individualization, enabling operations, and application-specific. Availability is discussed in media and platforms, ranging from mobile solutions to leading-edge supercomputers, with three accessibility levels. The major application-wise shift has been from research to clinical practice, particularly in stereotactic and functional neurosurgery, although clinical applications are still lagging behind the atlas content progress. Atlas functionality also has been relatively neglected until recently, as the management of brain data explosion requires powerful tools. We suggest that the future human brain atlas-related research and development activities shall be founded on and benefit from a standard framework containing the core virtual brain model cum the brain atlas platform general architecture.
Collapse
Affiliation(s)
- Wieslaw L Nowinski
- John Paul II Center for Virtual Anatomy and Surgical Simulation, University of Cardinal Stefan Wyszynski, Woycickiego 1/3, Block 12, room 1220, 01-938, Warsaw, Poland.
| |
Collapse
|
10
|
Cui Y, Zhao S, Chen Y, Han J, Guo L, Xie L, Liu T. Modeling Brain Diverse and Complex Hemodynamic Response Patterns via Deep Recurrent Autoencoder. IEEE Trans Cogn Dev Syst 2020. [DOI: 10.1109/tcds.2019.2949195] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
11
|
Vu H, Kim HC, Jung M, Lee JH. fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations. Neuroimage 2020; 223:117328. [PMID: 32896633 DOI: 10.1016/j.neuroimage.2020.117328] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2019] [Revised: 07/16/2020] [Accepted: 08/20/2020] [Indexed: 12/22/2022] Open
Abstract
Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.
Collapse
Affiliation(s)
- Hanh Vu
- Department of Brain and Cognitive Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul 02841, Republic of Korea
| | - Hyun-Chul Kim
- Department of Brain and Cognitive Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul 02841, Republic of Korea
| | - Minyoung Jung
- Department of Brain and Cognitive Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul 02841, Republic of Korea
| | - Jong-Hwan Lee
- Department of Brain and Cognitive Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul 02841, Republic of Korea.
| |
Collapse
|
12
|
Peña-Solórzano CA, Albrecht DW, Bassed RB, Gillam J, Harris PC, Dimmock MR. Semi-supervised labelling of the femur in a whole-body post-mortem CT database using deep learning. Comput Biol Med 2020; 122:103797. [PMID: 32658723 DOI: 10.1016/j.compbiomed.2020.103797] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2020] [Revised: 04/29/2020] [Accepted: 04/29/2020] [Indexed: 01/16/2023]
Abstract
A deep learning pipeline was developed and used to localize and classify a variety of implants in the femur contained in whole-body post-mortem computed tomography (PMCT) scans. The results provide a proof-of-principle approach for labelling content not described in medical/autopsy reports. The pipeline, which incorporated residual networks and an autoencoder, was trained and tested using n = 450 full-body PMCT scans. For the localization component, Dice scores of 0.99, 0.96, and 0.98 and mean absolute errors of 3.2, 7.1, and 4.2 mm were obtained in the axial, coronal, and sagittal views, respectively. A regression analysis found the orientation of the implant to the scanner axis and also the relative positioning of extremities to be statistically significant factors. For the classification component, test cases were properly labelled as nail (N+), hip replacement (H+), knee replacement (K+) or without-implant (I-) with an accuracy >97%. The recall for I- and H+ cases was 1.00, but fell to 0.82 and 0.65 for cases with K+ and N+. This semi-automatic approach provides a generalized structure for image-based labelling of features, without requiring time-consuming segmentation.
Collapse
Affiliation(s)
- C A Peña-Solórzano
- Department of Medical Imaging and Radiation Sciences, Monash University, Wellington Rd, Clayton, Melbourne, VIC, 3800, Australia.
| | - D W Albrecht
- Clayton School of Information Technology, Monash University, Wellington Rd, Clayton, Melbourne, VIC, 3800, Australia.
| | - R B Bassed
- Victorian Institute of Forensic Medicine, 57-83 Kavanagh St., Southbank, Melbourne, VIC, 3006, Australia; Department of Forensic Medicine, Monash University, Wellington Rd, Clayton, Melbourne, VIC, 3800, Australia.
| | - J Gillam
- Land Division, Defence Science and Technology Group, Fishermans Bend, Melbourne, VIC, 3207, Australia.
| | - P C Harris
- The Royal Children's Hospital Melbourne, 50 Flemington Road, Parkville, Melbourne, VIC, 3052, Australia; Department of Orthopaedic Surgery, Western Health, Footscray Hospital, Gordon St, Footscray, Melbourne, VIC, 3011, Australia.
| | - M R Dimmock
- Department of Medical Imaging and Radiation Sciences, Monash University, Wellington Rd, Clayton, Melbourne, VIC, 3800, Australia.
| |
Collapse
|
13
|
Dong Q, Ge F, Ning Q, Zhao Y, Lv J, Huang H, Yuan J, Jiang X, Shen D, Liu T. Modeling Hierarchical Brain Networks via Volumetric Sparse Deep Belief Network. IEEE Trans Biomed Eng 2020; 67:1739-1748. [DOI: 10.1109/tbme.2019.2945231] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
14
|
Cui Y, Zhao S, Wang H, Xie L, Chen Y, Han J, Guo L, Zhou F, Liu T. Identifying Brain Networks at Multiple Time Scales via Deep Recurrent Neural Network. IEEE J Biomed Health Inform 2019; 23:2515-2525. [PMID: 30475739 PMCID: PMC6914656 DOI: 10.1109/jbhi.2018.2882885] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
For decades, task functional magnetic resonance imaging has been a powerful noninvasive tool to explore the organizational architecture of human brain function. Researchers have developed a variety of brain network analysis methods for task fMRI data, including the general linear model, independent component analysis, and sparse representation methods. However, these shallow models are limited in faithful reconstruction and modeling of the hierarchical and temporal structures of brain networks, as demonstrated in more and more studies. Recently, recurrent neural networks (RNNs) exhibit great ability of modeling hierarchical and temporal dependence features in the machine learning field, which might be suitable for task fMRI data modeling. To explore such possible advantages of RNNs for task fMRI data, we propose a novel framework of a deep recurrent neural network (DRNN) to model the functional brain networks from task fMRI data. Experimental results on the motor task fMRI data of Human Connectome Project 900 subjects release demonstrated that the proposed DRNN can not only faithfully reconstruct functional brain networks, but also identify more meaningful brain networks with multiple time scales which are overlooked by traditional shallow models. In general, this work provides an effective and powerful approach to identifying functional brain networks at multiple time scales from task fMRI data.
Collapse
Affiliation(s)
- Yan Cui
- College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, 310027, China
| | - Shijie Zhao
- School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
| | - Han Wang
- College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, 310027, China
| | - Li Xie
- College of Biomedical Engineering & Instrument Science, and the State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou, 310027, China
| | - Yaowu Chen
- College of Biomedical Engineering & Instrument Science, and Zhejiang Provincial Key Laboratory for Network Multimedia Technologies, Zhejiang University, Hangzhou, 310027, China
| | - Junwei Han
- School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
| | - Lei Guo
- School of Automation, Northwestern Polytechnical University, Xi’an, 710072, China
| | - Fan Zhou
- College of Biomedical Engineering & Instrument Science, Zhejiang University, and the Zhejiang University Embedded System Engineering Research Center, Ministry of Education of China, Hangzhou, 310027, China
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, 30602, USA
| |
Collapse
|
15
|
Li X, Guo N, Li Q. Functional Neuroimaging in the New Era of Big Data. GENOMICS, PROTEOMICS & BIOINFORMATICS 2019; 17:393-401. [PMID: 31809864 PMCID: PMC6943787 DOI: 10.1016/j.gpb.2018.11.005] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/01/2018] [Revised: 09/17/2018] [Accepted: 12/25/2018] [Indexed: 12/15/2022]
Abstract
The field of functional neuroimaging has substantially advanced as a big data science in the past decade, thanks to international collaborative projects and community efforts. Here we conducted a literature review on functional neuroimaging, with focus on three general challenges in big data tasks: data collection and sharing, data infrastructure construction, and data analysis methods. The review covers a wide range of literature types including perspectives, database descriptions, methodology developments, and technical details. We show how each of the challenges was proposed and addressed, and how these solutions formed the three core foundations for the functional neuroimaging as a big data science and helped to build the current data-rich and data-driven community. Furthermore, based on our review of recent literature on the upcoming challenges and opportunities toward future scientific discoveries, we envisioned that the functional neuroimaging community needs to advance from the current foundations to better data integration infrastructure, methodology development toward improved learning capability, and multi-discipline translational research framework for this new era of big data.
Collapse
Affiliation(s)
- Xiang Li
- Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA
| | - Ning Guo
- Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA
| | - Quanzheng Li
- Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA.
| |
Collapse
|
16
|
Automatic recognition of holistic functional brain networks using iteratively optimized convolutional neural networks (IO-CNN) with weak label initialization. Med Image Anal 2019; 47:111-126. [PMID: 29705574 DOI: 10.1016/j.media.2018.04.002] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2017] [Revised: 04/04/2018] [Accepted: 04/07/2018] [Indexed: 01/09/2023]
Abstract
fMRI data decomposition techniques have advanced significantly from shallow models such as Independent Component Analysis (ICA) and Sparse Coding and Dictionary Learning (SCDL) to deep learning models such Deep Belief Networks (DBN) and Convolutional Autoencoder (DCAE). However, interpretations of those decomposed networks are still open questions due to the lack of functional brain atlases, no correspondence across decomposed or reconstructed networks across different subjects, and significant individual variabilities. Recent studies showed that deep learning, especially deep convolutional neural networks (CNN), has extraordinary ability of accommodating spatial object patterns, e.g., our recent works using 3D CNN for fMRI-derived network classifications achieved high accuracy with a remarkable tolerance for mistakenly labelled training brain networks. However, the training data preparation is one of the biggest obstacles in these supervised deep learning models for functional brain network map recognitions, since manual labelling requires tedious and time-consuming labours which will sometimes even introduce label mistakes. Especially for mapping functional networks in large scale datasets such as hundreds of thousands of brain networks used in this paper, the manual labelling method will become almost infeasible. In response, in this work, we tackled both the network recognition and training data labelling tasks by proposing a new iteratively optimized deep learning CNN (IO-CNN) framework with an automatic weak label initialization, which enables the functional brain networks recognition task to a fully automatic large-scale classification procedure. Our extensive experiments based on ABIDE-II 1099 brains' fMRI data showed the great promise of our IO-CNN framework.
Collapse
|
17
|
Zhang S, Dong Q, Zhang W, Huang H, Zhu D, Liu T. Discovering hierarchical common brain networks via multimodal deep belief network. Med Image Anal 2019; 54:238-252. [PMID: 30954851 PMCID: PMC6487231 DOI: 10.1016/j.media.2019.03.011] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2018] [Revised: 02/04/2019] [Accepted: 03/27/2019] [Indexed: 01/08/2023]
Abstract
Studying a common architecture reflecting both brain's structural and functional organizations across individuals and populations in a hierarchical way has been of significant interest in the brain mapping field. Recently, deep learning models exhibited ability in extracting meaningful hierarchical structures from brain imaging data, e.g., fMRI and DTI. However, deep learning models have been rarely used to explore the relation between brain structure and function yet. In this paper, we proposed a novel multimodal deep believe network (DBN) model to discover and quantitatively represent the hierarchical organizations of common and consistent brain networks from both fMRI and DTI data. A prominent characteristic of DBN is that it is capable of extracting meaningful features from complex neuroimaging data with a hierarchical manner. With our proposed DBN model, three hierarchical layers with hundreds of common and consistent brain networks across individual brains are successfully constructed through learning a large dimension of representative features from fMRI/DTI data.
Collapse
Affiliation(s)
- Shu Zhang
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| | - Qinglin Dong
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| | - Wei Zhang
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA
| | - Heng Huang
- School of Automation, Northwestern Polytechnical University, Xi'an, 710072, China
| | - Dajiang Zhu
- The University of Texas at Arlington, Arlington, TX 76010, USA
| | - Tianming Liu
- Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA.
| |
Collapse
|
18
|
Wen D, Wei Z, Zhou Y, Li G, Zhang X, Han W. Deep Learning Methods to Process fMRI Data and Their Application in the Diagnosis of Cognitive Impairment: A Brief Overview and Our Opinion. Front Neuroinform 2018; 12:23. [PMID: 29755334 PMCID: PMC5932168 DOI: 10.3389/fninf.2018.00023] [Citation(s) in RCA: 39] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2018] [Accepted: 04/11/2018] [Indexed: 01/18/2023] Open
Affiliation(s)
- Dong Wen
- Department of Software Engineering, School of Information Science and Engineering, Yanshan University, Qinhuangdao, China.,The Key Laboratory of Software Engineering of Hebei Province, Yanshan University, Qinhuangdao, China
| | - Zhenhao Wei
- Department of Software Engineering, School of Information Science and Engineering, Yanshan University, Qinhuangdao, China.,The Key Laboratory of Software Engineering of Hebei Province, Yanshan University, Qinhuangdao, China
| | - Yanhong Zhou
- Department of Computer Science and Technology, School of Mathematics and Information Science and Technology, Hebei Normal University of Science and Technology, Qinhuangdao, China
| | - Guolin Li
- Department of Educational Technology, College of Education, Hebei Normal University of Science and Technology, Qinhuangdao, China
| | - Xu Zhang
- Department of Software Engineering, School of Information Science and Engineering, Yanshan University, Qinhuangdao, China.,The Key Laboratory of Software Engineering of Hebei Province, Yanshan University, Qinhuangdao, China
| | - Wei Han
- Department of Software Engineering, School of Information Science and Engineering, Yanshan University, Qinhuangdao, China.,The Key Laboratory of Software Engineering of Hebei Province, Yanshan University, Qinhuangdao, China
| |
Collapse
|
19
|
Zeng LL, Wang H, Hu P, Yang B, Pu W, Shen H, Chen X, Liu Z, Yin H, Tan Q, Wang K, Hu D. Multi-Site Diagnostic Classification of Schizophrenia Using Discriminant Deep Learning with Functional Connectivity MRI. EBioMedicine 2018; 30:74-85. [PMID: 29622496 PMCID: PMC5952341 DOI: 10.1016/j.ebiom.2018.03.017] [Citation(s) in RCA: 148] [Impact Index Per Article: 24.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2018] [Revised: 03/06/2018] [Accepted: 03/16/2018] [Indexed: 01/10/2023] Open
Abstract
BACKGROUND A lack of a sufficiently large sample at single sites causes poor generalizability in automatic diagnosis classification of heterogeneous psychiatric disorders such as schizophrenia based on brain imaging scans. Advanced deep learning methods may be capable of learning subtle hidden patterns from high dimensional imaging data, overcome potential site-related variation, and achieve reproducible cross-site classification. However, deep learning-based cross-site transfer classification, despite less imaging site-specificity and more generalizability of diagnostic models, has not been investigated in schizophrenia. METHODS A large multi-site functional MRI sample (n = 734, including 357 schizophrenic patients from seven imaging resources) was collected, and a deep discriminant autoencoder network, aimed at learning imaging site-shared functional connectivity features, was developed to discriminate schizophrenic individuals from healthy controls. FINDINGS Accuracies of approximately 85·0% and 81·0% were obtained in multi-site pooling classification and leave-site-out transfer classification, respectively. The learned functional connectivity features revealed dysregulation of the cortical-striatal-cerebellar circuit in schizophrenia, and the most discriminating functional connections were primarily located within and across the default, salience, and control networks. INTERPRETATION The findings imply that dysfunctional integration of the cortical-striatal-cerebellar circuit across the default, salience, and control networks may play an important role in the "disconnectivity" model underlying the pathophysiology of schizophrenia. The proposed discriminant deep learning method may be capable of learning reliable connectome patterns and help in understanding the pathophysiology and achieving accurate prediction of schizophrenia across multiple independent imaging sites.
Collapse
Affiliation(s)
- Ling-Li Zeng
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, China
| | - Huaning Wang
- Department of Psychiatry, Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Panpan Hu
- Department of Neurology, The First Affiliated Hospital of Anhui Medical University, Hefei, China
| | - Bo Yang
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, China
| | - Weidan Pu
- Medical Psychological Center, Second Xiangya Hospital, Central South University, Changsha, China
| | - Hui Shen
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, China
| | - Xingui Chen
- Department of Neurology, The First Affiliated Hospital of Anhui Medical University, Hefei, China
| | - Zhening Liu
- Mental Health Institute, Second Xiangya Hospital, Central South University, Changsha, China
| | - Hong Yin
- Department of Radiology, Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Qingrong Tan
- Department of Psychiatry, Xijing Hospital, Fourth Military Medical University, Xi'an, China.
| | - Kai Wang
- Department of Neurology, The First Affiliated Hospital of Anhui Medical University, Hefei, China.
| | - Dewen Hu
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, China.
| |
Collapse
|
20
|
Zhao S, Han J, Jiang X, Huang H, Liu H, Lv J, Guo L, Liu T. Decoding Auditory Saliency from Brain Activity Patterns during Free Listening to Naturalistic Audio Excerpts. Neuroinformatics 2018; 16:309-324. [DOI: 10.1007/s12021-018-9358-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|
21
|
Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN). MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION – MICCAI 2018 2018. [DOI: 10.1007/978-3-030-00931-1_21] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|