Volumetric brain MRI signatures of heart failure with preserved ejection fraction in the setting of dementia

Heart failure with preserved ejection fraction (HFpEF) is an important, emerging risk factor for dementia, but it is not clear whether HFpEF contributes to a specific pattern of neuroanatomical changes in dementia. A major challenge to studying this is the relative paucity of datasets of patients with dementia, with/without HFpEF, and relevant neuroimaging. We sought to demonstrate the feasibility of using modern data mining tools to create and analyze clinical imaging datasets and identify the neuroanatomical signature of HFpEF-associated dementia. We leveraged the bioinformatics tools at Vanderbilt University Medical Center to identify patients with a diagnosis of dementia with and without comorbid HFpEF using the electronic health record. We identified high resolution, clinically-acquired neuroimaging data on 30 dementia patients with HFpEF (age 76.9 ± 8.12 years, 61% female) as well as 301 age- and sex-matched patients with dementia but without HFpEF to serve as comparators (age 76.2 ± 8.52 years, 60% female). We used automated image processing pipelines to parcellate the brain into 132 structures and quantify their volume. We found six regions with significant atrophy associated with HFpEF: accumbens area, amygdala, posterior insula, anterior orbital gyrus, angular gyrus, and cerebellar white matter. There were no regions with atrophy inversely associated with HFpEF. Patients with dementia and HFpEF have a distinct neuroimaging signature compared to patients with dementia only. Five of the six regions identified in are in the temporo-parietal region of the brain. Future studies should investigate mechanisms of injury associated with cerebrovascular disease leading to subsequent brain atrophy.

Phenotyping Down syndrome: discovery and predictive modelling with electronic medical records

BACKGROUND

Individuals with Down syndrome (DS) have a heightened risk for various co-occurring health conditions, including congenital heart disease (CHD). In this two-part study, electronic medical records (EMRs) were leveraged to examine co-occurring health conditions among individuals with DS (Study 1) and to investigate health conditions linked to surgical intervention among DS cases with CHD (Study 2).

METHODS

De-identified EMRs were acquired from Vanderbilt University Medical Center and facilitated creating a cohort of N = 2282 DS cases (55% females), along with comparison groups for each study. In Study 1, DS cases were one-by-two sex and age matched with samples of case-controls and of individuals with other intellectual and developmental difficulties (IDDs). The phenome-disease association study (PheDAS) strategy was employed to reveal co-occurring health conditions in DS versus comparison groups, which were then ranked for how often they are discussed in relation to DS using the PubMed database and Novelty Finding Index. In Study 2, a subset of DS individuals with CHD [N = 1098 (48%)] were identified to create longitudinal data for N = 204 cases with surgical intervention (19%) versus 204 case-controls. Data were included in predictive models and assessed which model-based health conditions, when more prevalent, would increase the likelihood of surgical intervention.

RESULTS

In Study 1, relative to case-controls and those with other IDDs, co-occurring health conditions among individuals with DS were confirmed to include heart failure, pulmonary heart disease, atrioventricular block, heart transplant/surgery and primary pulmonary hypertension (circulatory); hypothyroidism (endocrine/metabolic); and speech and language disorder and Alzheimer's disease (neurological/mental). Findings also revealed more versus less prevalent co-occurring health conditions in individuals with DS when comparing with those with other IDDs. Findings with high Novelty Finding Index were abnormal electrocardiogram, non-rheumatic aortic valve disorders and heart failure (circulatory); acid-base balance disorder (endocrine/metabolism); and abnormal blood chemistry (symptoms). In Study 2, the predictive models revealed that among individuals with DS and CHD, presence of health conditions such as congestive heart failure (circulatory), valvular heart disease and cardiac shunt (congenital), and pleural effusion and pulmonary collapse (respiratory) were associated with increased likelihood of surgical intervention.

CONCLUSIONS

Research efforts using EMRs and rigorous statistical methods could shed light on the complexity in health profile among individuals with DS and other IDDs and motivate precision-care development.

Comprehensive behavioral and physiologic assessment of peripheral and central auditory function in individuals with mild traumatic brain injury

Auditory complaints are frequently reported by individuals with mild traumatic brain injury (mTBI) yet remain difficult to detect in the absence of clinically significant hearing loss. This highlights a growing need to identify sensitive indices of auditory-related mTBI pathophysiology beyond pure-tone thresholds for improved hearing healthcare diagnosis and treatment. Given the heterogeneity of mTBI etiology and the diverse peripheral and central processes required for normal auditory function, the present study sought to determine the audiologic assessments sensitive to mTBI pathophysiology at the group level using a well-rounded test battery of both peripheral and central auditory system function. This test battery included pure-tone detection thresholds, word understanding in quiet, sentence understanding in noise, distortion product otoacoustic emissions (DPOAEs), middle-ear muscle reflexes (MEMRs), and auditory evoked potentials (AEPs), including auditory brainstem responses (ABRs), middle latency responses (MLRs), and late latency responses (LLRs). Each participant also received magnetic resonance imaging (MRI). Compared to the control group, we found that individuals with mTBI had reduced DPOAE amplitudes that revealed a compound effect of age, elevated MEMR thresholds for an ipsilateral broadband noise elicitor, longer ABR Wave I latencies for click and 4 kHz tone burst elicitors, longer ABR Wave III latencies for 4 kHz tone bursts, larger MLR Na and Nb amplitudes, smaller MLR Pb amplitudes, longer MLR Pa latencies, and smaller LLR N1 amplitudes for older individuals with mTBI. Further, mTBI individuals with combined hearing difficulty and noise sensitivity had a greater number of deficits on thalamic and cortical AEP measures compared to those with only one/no self-reported auditory symptoms. This finding was corroborated with MRI, which revealed significant structural differences in the auditory cortical areas of mTBI participants who reported combined hearing difficulty and noise sensitivity, including an enlargement of left transverse temporal gyrus (TTG) and bilateral planum polare (PP). These findings highlight the need for continued investigations toward identifying individualized audiologic assessments and treatments that are sensitive to mTBI pathophysiology.

pyPheWAS Explorer: a visualization tool for exploratory analysis of phenome-disease associations

Objective

To enable interactive visualization of phenome-wide association studies (PheWAS) on electronic health records (EHR).

Materials and Methods

Current PheWAS technologies require familiarity with command-line interfaces and lack end-to-end data visualizations. pyPheWAS Explorer allows users to examine group variables, test assumptions, design PheWAS models, and evaluate results in a streamlined graphical interface.

Results

A cohort of attention deficit hyperactivity disorder (ADHD) subjects and matched non-ADHD controls is examined. pyPheWAS Explorer is used to build a PheWAS model including sex and deprivation index as covariates, and the Explorer's result visualization for this model reveals known ADHD comorbidities.

Discussion

pyPheWAS Explorer may be used to rapidly investigate potentially novel EHR associations. Broader applications include deployment for clinical experts and preliminary exploration tools for institutional EHR repositories.

Conclusion

pyPheWAS Explorer provides a seamless graphical interface for designing, executing, and analyzing PheWAS experiments, emphasizing exploratory analysis of regression types and covariate selection.

Health Conditions Associated with Alzheimer's Disease and Vascular Dementia

OBJECTIVE

We examined medical records to determine health conditions associated with dementia at varied intervals prior to dementia diagnosis in participants from the Baltimore Longitudinal Study of Aging (BLSA).

METHODS

Data were available for 347 Alzheimer's disease (AD), 76 vascular dementia (VaD), and 811 control participants without dementia. Logistic regressions were performed associating International Classification of Diseases, 9th Revision (ICD-9) health codes with dementia status across all time points, at 5 and 1 year(s) prior to dementia diagnosis, and at the year of diagnosis, controlling for age, sex, and follow-up length of the medical record.

RESULTS

In AD, the earliest and most consistent associations across all time points included depression, erectile dysfunction, gait abnormalities, hearing loss, and nervous and musculoskeletal symptoms. Cardiomegaly, urinary incontinence, non-epithelial skin cancer, and pneumonia were not significant until 1 year before dementia diagnosis. In VaD, the earliest and most consistent associations across all time points included abnormal electrocardiogram (EKG), cardiac dysrhythmias, cerebrovascular disease, non-epithelial skin cancer, depression, and hearing loss. Atrial fibrillation, occlusion of cerebral arteries, essential tremor, and abnormal reflexes were not significant until 1 year before dementia diagnosis.

INTERPRETATION

These findings suggest that some health conditions are associated with future dementia beginning at least 5 years before dementia diagnosis and are consistently seen over time, while others only reach significance closer to the date of diagnosis. These results also show that there are both shared and distinctive health conditions associated with AD and VaD. These results reinforce the need for medical intervention and treatment to lessen the impact of health comorbidities in the aging population. ANN NEUROL 2023;93:805-818.

Convolutional-recurrent neural networks approximate diffusion tractography from T1-weighted MRI and associated anatomical context

Diffusion MRI (dMRI) streamline tractography is the gold-standard for in vivo estimation of white matter (WM) pathways in the brain. However, the high angular resolution dMRI acquisitions capable of fitting the microstructural models needed for tractography are often time-consuming and not routinely collected clinically, restricting the scope of tractography analyses. To address this limitation, we build on recent advances in deep learning which have demonstrated that streamline propagation can be learned from dMRI directly without traditional model fitting. Specifically, we propose learning the streamline propagator from T1w MRI to facilitate arbitrary tractography analyses when dMRI is unavailable. To do so, we present a novel convolutional-recurrent neural network (CoRNN) trained in a teacher-student framework that leverages T1w MRI, associated anatomical context, and streamline memory from data acquired for the Human Connectome Project. We characterize our approach under two common tractography paradigms, WM bundle analysis and structural connectomics, and find approximately a 5-15% difference between measures computed from streamlines generated with our approach and those generated using traditional dMRI tractography. When placed in the literature, these results suggest that the accuracy of WM measures computed from T1w MRI with our method is on the level of scan-rescan dMRI variability and raise an important question: is tractography truly a microstructural phenomenon, or has dMRI merely facilitated its discovery and implementation?

Batch size: go big or go home? Counterintuitive improvement in medical autoencoders with smaller batch size

Batch size is a key hyperparameter in training deep learning models. Conventional wisdom suggests larger batches produce improved model performance. Here we present evidence to the contrary, particularly when using autoencoders to derive meaningful latent spaces from data with spatially global similarities and local differences, such as electronic health records (EHR) and medical imaging. We investigate batch size effects in both EHR data from the Baltimore Longitudinal Study of Aging and medical imaging data from the multimodal brain tumor segmentation (BraTS) challenge. We train fully connected and convolutional autoencoders to compress the EHR and imaging input spaces, respectively, into 32-dimensional latent spaces via reconstruction losses for various batch sizes between 1 and 100. Under the same hyperparameter configurations, smaller batches improve loss performance for both datasets. Additionally, latent spaces derived by autoencoders with smaller batches capture more biologically meaningful information. Qualitatively, we visualize 2-dimensional projections of the latent spaces and find that with smaller batches the EHR network better separates the sex of the individuals, and the imaging network better captures the right-left laterality of tumors. Quantitatively, the analogous sex classification and laterality regressions using the latent spaces demonstrate statistically significant improvements in performance at smaller batch sizes. Finally, we find improved individual variation locally in visualizations of representative data reconstructions at lower batch sizes. Taken together, these results suggest that smaller batch sizes should be considered when designing autoencoders to extract meaningful latent spaces among EHR and medical imaging data driven by global similarities and local variation.

Predicting Crohn's disease severity in the colon using mixed cell nucleus density from pseudo labels

Crohn's disease (CD) is a debilitating inflammatory bowel disease with no known cure. Computational analysis of hematoxylin and eosin (H&E) stained colon biopsy whole slide images (WSIs) from CD patients provides the opportunity to discover unknown and complex relationships between tissue cellular features and disease severity. While there have been works using cell nuclei-derived features for predicting slide-level traits, this has not been performed on CD H&E WSIs for classifying normal tissue from CD patients vs active CD and assessing slide label-predictive performance while using both separate and combined information from pseudo-segmentation labels of nuclei from neutrophils, eosinophils, epithelial cells, lymphocytes, plasma cells, and connective cells. We used 413 WSIs of CD patient biopsies and calculated normalized histograms of nucleus density for the six cell classes for each WSI. We used a support vector machine to classify the truncated singular value decomposition representations of the normalized histograms as normal or active CD with four-fold cross-validation in rounds where nucleus types were first compared individually, the best was selected, and further types were added each round. We found that neutrophils were the most predictive individual nucleus type, with an AUC of 0.92 ± 0.0003 on the withheld test set. Adding information improved cross-validation performance for the first two rounds and on the withheld test set for the first three rounds, though performance metrics did not increase substantially beyond when neutrophils were used alone.

Joint independent component analysis for hypothesizing spatiotemporal relationships between longitudinal gray and white matter changes in preclinical Alzheimer's disease

Characterizing relationships between gray matter (GM) and white matter (WM) in early Alzheimer's disease (AD) would improve understanding of how and when AD impacts the brain. However, modeling these relationships across brain regions and longitudinally remains a challenge. Thus, we propose extending joint independent component analysis (jICA) into spatiotemporal modeling of regional cortical thickness and WM bundle volumes leveraging multimodal MRI. We jointly characterize these GM and WM features in a normal aging (n=316) and an age- and sex-matched preclinical AD cohort (n=81) at each of two imaging sessions spaced three years apart, training on the normal aging population in cross-validation and interrogating the preclinical AD cohort. We find this joint model identifies reproducible, longitudinal changes in GM and WM between the two imaging sessions and that these changes are associated with preclinical AD and are plausible considering the literature. We compare this joint model to two focused models: (1) GM features at the first session and WM at the second and (2) vice versa. The joint model identifies components that correlate poorly with those from the focused models, suggesting the different models resolve different patterns. We find the strength of association with preclinical AD is improved in the GM to WM model, which supports the hypothesis that medial temporal and frontal thinning precedes volume loss in the uncinate fasciculus and inferior anterior-posterior association fibers. These results suggest that jICA effectively generates spatiotemporal hypotheses about GM and WM in preclinical AD, especially when specific intermodality relationships are considered a priori.

pyPheWAS: A Phenome-Disease Association Tool for Electronic Medical Record Analysis

Along with the increasing availability of electronic medical record (EMR) data, phenome-wide association studies (PheWAS) and phenome-disease association studies (PheDAS) have become a prominent, first-line method of analysis for uncovering the secrets of EMR. Despite this recent growth, there is a lack of approachable software tools for conducting these analyses on large-scale EMR cohorts. In this article, we introduce pyPheWAS, an open-source python package for conducting PheDAS and related analyses. This toolkit includes 1) data preparation, such as cohort censoring and age-matching; 2) traditional PheDAS analysis of ICD-9 and ICD-10 billing codes; 3) PheDAS analysis applied to a novel EMR phenotype mapping: current procedural terminology (CPT) codes; and 4) novelty analysis of significant disease-phenotype associations found through PheDAS. The pyPheWAS toolkit is approachable and comprehensive, encapsulating data prep through result visualization all within a simple command-line interface. The toolkit is designed for the ever-growing scale of available EMR data, with the ability to analyze cohorts of 100,000 + patients in less than 2 h. Through a case study of Down Syndrome and other intellectual developmental disabilities, we demonstrate the ability of pyPheWAS to discover both known and potentially novel disease-phenotype associations across different experiment designs and disease groups. The software and user documentation are available in open source at https://github.com/MASILab/pyPheWAS .

MASiVar: Multisite, multiscanner, and multisubject acquisitions for studying variability in diffusion weighted MRI

PURPOSE

Diffusion-weighted imaging allows investigators to identify structural, microstructural, and connectivity-based differences between subjects, but variability due to session and scanner biases is a challenge.

METHODS

To investigate DWI variability, we present MASiVar, a multisite data set consisting of 319 diffusion scans acquired at 3 T from b = 1000 to 3000 s/mm2 across 14 healthy adults, 83 healthy children (5 to 8 years), three sites, and four scanners as a publicly available, preprocessed, and de-identified data set. With the adult data, we demonstrate the capacity of MASiVar to simultaneously quantify the intrasession, intersession, interscanner, and intersubject variability of four common DWI processing approaches: (1) a tensor signal representation, (2) a multi-compartment neurite orientation dispersion and density model, (3) white-matter bundle segmentation, and (4) structural connectomics. Respectively, we evaluate region-wise fractional anisotropy, mean diffusivity, and principal eigenvector; region-wise CSF volume fraction, intracellular volume fraction, and orientation dispersion index; bundle-wise shape, volume, fractional anisotropy, and length; and whole connectome correlation and maximized modularity, global efficiency, and characteristic path length.

RESULTS

We plot the variability in these measures at each level and find that it consistently increases with intrasession to intersession to interscanner to intersubject effects across all processing approaches and that sometimes interscanner variability can approach intersubject variability.

CONCLUSIONS

This study demonstrates the potential of MASiVar to more globally investigate DWI variability across multiple levels and processing approaches simultaneously and suggests harmonization between scanners for multisite analyses should be considered before inference of group differences on subjects.

Validation of Group-wise Registration for Surface-based Functional MRI Analysis

Resting-state functional MRI (rsfMRI) provides important information for studying and mapping the activities and functions of the brain. Conventionally, rsfMRIs are often registered to structural images in the Euclidean space without considering cortical surface geometry. Meanwhile, a surface-based representation offers a relaxed coordinate chart, but this still requires surface registration for group-wise data analysis. In this work, we investigate the performance of two existing surface registration methods in a surface-based rsfMRI analysis framework: FreeSurfer and Hierarchical Spherical Deformation (HSD). To minimize registration bias, we establish shape correspondence using both methods in a group-wise manner that estimates the unbiased average of a given cohort. To evaluate their performance, we focus on neuroanatomical alignment as well as the amount of distortion that can potentially bias surface tessellation for secondary level rsfMRI data analyses. In the pilot analysis, we examine a single timepoint of imaging data from 100 subjects out of an aging cohort. Overall, HSD establishes improved shape correspondence with reduced mean curvature deviation (10.94% less on average per subject, paired t-test: p <10-10) and reduced registration distortion (FreeSurfer: average 41.91% distortion per subject, HSD: 18.63%, paired t-test: p <10-10). Furthermore, HSD introduces less distortion than FreeSurfer in the areas identified in the individual components that were extracted by surface-based independent component analysis (ICA) after spatial smoothing and time series normalization. Consequently, we show that FreeSurfer capture individual components with globally similar but locally different patterns in ICA in visual inspection.

Joint analysis of structural connectivity and cortical surface features: correlates with mild traumatic brain injury

Mild traumatic brain injury (mTBI) is a complex syndrome that affects up to 600 per 100,000 individuals, with a particular concentration among military personnel. About half of all mTBI patients experience a diverse array of chronic symptoms which persist long after the acute injury. Hence, there is an urgent need for better understanding of the white matter and gray matter pathologies associated with mTBI to map which specific brain systems are impacted and identify courses of intervention. Previous works have linked mTBI to disruptions in white matter pathways and cortical surface abnormalities. Herein, we examine these hypothesized links in an exploratory study of joint structural connectivity and cortical surface changes associated with mTBI and its chronic symptoms. Briefly, we consider a cohort of 12 mTBI and 26 control subjects. A set of 588 cortical surface metrics and 4,753 structural connectivity metrics were extracted from cortical surface regions and diffusion weighted magnetic resonance imaging in each subject. Principal component analysis (PCA) was used to reduce the dimensionality of each metric set. We then applied independent component analysis (ICA) both to each PCA space individually and together in a joint ICA approach. We identified a stable independent component across the connectivity-only and joint ICAs which presented significant group differences in subject loadings (p<0.05, corrected). Additionally, we found that two mTBI symptoms, slowed thinking and forgetfulness, were significantly correlated (p<0.05, corrected) with mTBI subject loadings in a surface-only ICA. These surface-only loadings captured an increase in bilateral cortical thickness.

Joint cortical surface and structural connectivity analysis of Alzheimer's Disease

Prior neuroimaging studies have demonstrated isolated structural and connectivity changes in the brain due to Alzheimer's Disease (AD). However, how these changes relate to each other is not well understood. We present a preliminary study to begin to fill this gap by leveraging joint independent component analysis (jICA). We explore how jICA performs in an analysis of T1 and diffusion weighted MRI by characterizing the joint changes of complex cortical surface and structural connectivity metrics in AD in subjects from the Baltimore Longitudinal Study of Aging. We calculate 588 region-based cortical metrics and 4,753 fractional anisotropy-based connectivity metrics and project them into a low-dimensional manifold with principal component analysis. We perform jICA on the manifold and subsequently backproject the independent components to the original data space. We demonstrate component stability with 3-fold cross validation and find differential component loadings between 776 cognitively unimpaired control subjects and 23 with AD that generalizes across folds. In addition, we perform the same analysis on the surface and connectivity metrics separately and find that the joint approach identifies both novel and similar components to the separate approaches. To illustrate the joint approach's primary utility, we provide an example hypothesis for how surface and connectivity components may vary together with AD. These preliminary results suggest jointly varying independent cortical surface and structural connectivity components can be consistently extracted from MRI data and provide a data-driven way for generating novel hypotheses about AD that may not be captured by separate analyses.

Extracting 2D weak labels from volume labels using multiple instance learning in CT hemorrhage detection

Multiple instance learning (MIL) is a supervised learning methodology that aims to allow models to learn instance class labels from bag class labels, where a bag is defined to contain multiple instances. MIL is gaining traction for learning from weak labels but has not been widely applied to 3D medical imaging. MIL is well-suited to clinical CT acquisitions since (1) the highly anisotropic voxels hinder application of traditional 3D networks and (2) patch-based networks have limited ability to learn whole volume labels. In this work, we apply MIL with a deep convolutional neural network to identify whether clinical CT head image volumes possess one or more large hemorrhages (> 20cm3), resulting in a learned 2D model without the need for 2D slice annotations. Individual image volumes are considered separate bags, and the slices in each volume are instances. Such a framework sets the stage for incorporating information obtained in clinical reports to help train a 2D segmentation approach. Within this context, we evaluate the data requirements to enable generalization of MIL by varying the amount of training data. Our results show that a training size of at least 400 patient image volumes was needed to achieve accurate per-slice hemorrhage detection. Over a five-fold cross-validation, the leading model, which made use of the maximum number of training volumes, had an average true positive rate of 98.10%, an average true negative rate of 99.36%, and an average precision of 0.9698. The models have been made available along with source code1 to enabled continued exploration and adaption of MIL in CT neuroimaging.

MRI correlates of chronic symptoms in mild traumatic brain injury

Some veterans with a history of mild traumatic brain injury (mTBI) have reported experiencing auditory and visual dysfunction that persist beyond the acute phase of the incident. The etiology behind these symptoms is difficult to characterize, since mTBI is defined by negative imaging findings on current clinical imaging. There are several competing hypotheses that could explain functional deficits; one example is shear injury, which may manifest in diffusion-weighted magnetic resonance (MR) imaging (DWI). Herein, we explore this alternative hypothesis in a pilot study of multi-parametric MR imaging. Briefly, we consider a cohort of 8 mTBI patients relative to 22 control subjects using structural T1-weighted imaging (T1w) and connectivity with DWI. 1,344 metrics were extracted per subject from whole brain regions and connectivity patterns in sensory networks. For each set of imaging-derived metrics, the control subject metrics were embedded in a low-dimensional manifold with principal component analysis, after which mTBI subject metrics were projected into the same space. These manifolds were employed to train support vector machines (SVM) to classify subjects as controls or mTBI. Two of the SVMs trained achieved near-perfect accuracy averaged across four-fold cross-validation. Additionally, we present correlations between manifold dimensions and 22 self-reported mTBI symptoms and find that five principal components from the manifolds (one component from the T1w manifold and four components from the DWI manifold) are significantly correlated with symptoms (p<0.05, uncorrected). The novelty of this work is that the DWI and T1w imaging metrics seem to contain information critical for distinguishing between mTBI and control subjects. This work presents an analysis of the pilot phase of data collection of the Quantitative Evaluation of Visual and Auditory Dysfunction and Multi-Sensory Integration in Complex TBI Patients study and defines specific hypotheses to be tested in the full sample.

Montage based 3D Medical Image Retrieval from Traumatic Brain Injury Cohort using Deep Convolutional Neural Network

Brain imaging analysis on clinically acquired computed tomography (CT) is essential for the diagnosis, risk prediction of progression, and treatment of the structural phenotypes of traumatic brain injury (TBI). However, in real clinical imaging scenarios, entire body CT images (e.g., neck, abdomen, chest, pelvis) are typically captured along with whole brain CT scans. For instance, in a typical sample of clinical TBI imaging cohort, only ~15% of CT scans actually contain whole brain CT images suitable for volumetric brain analyses; the remaining are partial brain or non-brain images. Therefore, a manual image retrieval process is typically required to isolate the whole brain CT scans from the entire cohort. However, the manual image retrieval is time and resource consuming and even more difficult for the larger cohorts. To alleviate the manual efforts, in this paper we propose an automated 3D medical image retrieval pipeline, called deep montage-based image retrieval (dMIR), which performs classification on 2D montage images via a deep convolutional neural network. The novelty of the proposed method for image processing is to characterize the medical image retrieval task based on the montage images. In a cohort of 2000 clinically acquired TBI scans, 794 scans were used as training data, 206 scans were used as validation data, and the remaining 1000 scans were used as testing data. The proposed achieved accuracy=1.0, recall=1.0, precision=1.0, f1=1.0 for validation data, while achieved accuracy=0.988, recall=0.962, precision=0.962, f1=0.962 for testing data. Thus, the proposed dMIR is able to perform accurate CT whole brain image retrieval from large-scale clinical cohorts.