DeepNeuron: an open deep learning toolbox for neuron tracing

NeuroQuantify - An image analysis software for detection and quantification of neuron cells and neurite lengths using deep learning

BACKGROUND

The segmentation of cells and neurites in microscopy images of neuronal networks provides valuable quantitative information about neuron growth and neuronal differentiation, including the number of cells, neurites, neurite length and neurite orientation. This information is essential for assessing the development of neuronal networks in response to extracellular stimuli, which is useful for studying neuronal structures, for example, the study of neurodegenerative diseases and pharmaceuticals.

NEW METHOD

We have developed NeuroQuantify, an open-source software that uses deep learning to efficiently and quickly segment cells and neurites in phase contrast microscopy images.

RESULTS

NeuroQuantify offers several key features: (i) automatic detection of cells and neurites; (ii) post-processing of the images for the quantitative neurite length measurement based on segmentation of phase contrast microscopy images, and (iii) identification of neurite orientations.

COMPARISON WITH EXISTING METHODS

NeuroQuantify overcomes some of the limitations of existing methods in the automatic and accurate analysis of neuronal structures. It has been developed for phase contrast images rather than fluorescence images. In addition to typical functionality of cell counting, NeuroQuantify also detects and counts neurites, measures the neurite lengths, and produces the neurite orientation distribution.

CONCLUSIONS

We offer a valuable tool to assess network development rapidly and effectively. The user-friendly NeuroQuantify software can be installed and freely downloaded from GitHub at https://github.com/StanleyZ0528/neural-image-segmentation.

Efficient and Accurate Semi-Automatic Neuron Tracing with Extended Reality

Neuron tracing, alternately referred to as neuron reconstruction, is the procedure for extracting the digital representation of the three-dimensional neuronal morphology from stacks of microscopic images. Achieving accurate neuron tracing is critical for profiling the neuroanatomical structure at single-cell level and analyzing the neuronal circuits and projections at whole-brain scale. However, the process often demands substantial human involvement and represents a nontrivial task. Conventional solutions towards neuron tracing often contend with challenges such as non-intuitive user interactions, suboptimal data generation throughput, and ambiguous visualization. In this paper, we introduce a novel method that leverages the power of extended reality (XR) for intuitive and progressive semi-automatic neuron tracing in real time. In our method, we have defined a set of interactors for controllable and efficient interactions for neuron tracing in an immersive environment. We have also developed a GPU-accelerated automatic tracing algorithm that can generate updated neuron reconstruction in real time. In addition, we have built a visualizer for fast and improved visual experience, particularly when working with both volumetric images and 3D objects. Our method has been successfully implemented with one virtual reality (VR) headset and one augmented reality (AR) headset with satisfying results achieved. We also conducted two user studies and proved the effectiveness of the interactors and the efficiency of our method in comparison with other approaches for neuron tracing.

Gapr for large-scale collaborative single-neuron reconstruction

Whole-brain analysis of single-neuron morphology is crucial for unraveling the complex structure of the brain. However, large-scale neuron reconstruction from terabyte and even petabyte data of mammalian brains generated by state-of-the-art light microscopy is a daunting task. Here, we developed 'Gapr' (Gapr accelerates projectome reconstruction) that streamlines deep learning-based automatic reconstruction, 'automatic proofreading' that reduces human workloads at high-confidence sites, and high-throughput collaborative proofreading by crowd users through the Internet. Furthermore, Gapr offers a seamless user interface that ensures high proofreading speed per annotator, on-demand conversion for handling large datasets, flexible workflows tailored to diverse datasets and rigorous error tracking for quality control. Finally, we demonstrated Gapr's efficacy by reconstructing over 4,000 neurons in mouse brains, revealing the morphological diversity in cortical interneurons and hypothalamic neurons. Here, we present Gapr as a solution for large-scale single-neuron reconstruction projects.

High-throughput analysis of dendrite and axonal arbors reveals transcriptomic correlates of neuroanatomy

Neuronal anatomy is central to the organization and function of brain cell types. However, anatomical variability within apparently homogeneous populations of cells can obscure such insights. Here, we report large-scale automation of neuronal morphology reconstruction and analysis on a dataset of 813 inhibitory neurons characterized using the Patch-seq method, which enables measurement of multiple properties from individual neurons, including local morphology and transcriptional signature. We demonstrate that these automated reconstructions can be used in the same manner as manual reconstructions to understand the relationship between some, but not all, cellular properties used to define cell types. We uncover gene expression correlates of laminar innervation on multiple transcriptomically defined neuronal subclasses and types. In particular, our results reveal correlates of the variability in Layer 1 (L1) axonal innervation in a transcriptomically defined subpopulation of Martinotti cells in the adult mouse neocortex.

Connecto-informatics at the mesoscale: current advances in image processing and analysis for mapping the brain connectivity

Mapping neural connections within the brain has been a fundamental goal in neuroscience to understand better its functions and changes that follow aging and diseases. Developments in imaging technology, such as microscopy and labeling tools, have allowed researchers to visualize this connectivity through high-resolution brain-wide imaging. With this, image processing and analysis have become more crucial. However, despite the wealth of neural images generated, access to an integrated image processing and analysis pipeline to process these data is challenging due to scattered information on available tools and methods. To map the neural connections, registration to atlases and feature extraction through segmentation and signal detection are necessary. In this review, our goal is to provide an updated overview of recent advances in these image-processing methods, with a particular focus on fluorescent images of the mouse brain. Our goal is to outline a pathway toward an integrated image-processing pipeline tailored for connecto-informatics. An integrated workflow of these image processing will facilitate researchers' approach to mapping brain connectivity to better understand complex brain networks and their underlying brain functions. By highlighting the image-processing tools available for fluroscent imaging of the mouse brain, this review will contribute to a deeper grasp of connecto-informatics, paving the way for better comprehension of brain connectivity and its implications.

An unbiased, automated platform for scoring dopaminergic neurodegeneration in C. elegans

Caenorhabditis elegans (C. elegans) has served as a simple model organism to study dopaminergic neurodegeneration, as it enables quantitative analysis of cellular and sub-cellular morphologies in live animals. These isogenic nematodes have a rapid life cycle and transparent body, making high-throughput imaging and evaluation of fluorescently tagged neurons possible. However, the current state-of-the-art method for quantifying dopaminergic degeneration requires researchers to manually examine images and score dendrites into groups of varying levels of neurodegeneration severity, which is time consuming, subject to bias, and limited in data sensitivity. We aim to overcome the pitfalls of manual neuron scoring by developing an automated, unbiased image processing algorithm to quantify dopaminergic neurodegeneration in C. elegans. The algorithm can be used on images acquired with different microscopy setups and only requires two inputs: a maximum projection image of the four cephalic neurons in the C. elegans head and the pixel size of the user's camera. We validate the platform by detecting and quantifying neurodegeneration in nematodes exposed to rotenone, cold shock, and 6-hydroxydopamine using 63x epifluorescence, 63x confocal, and 40x epifluorescence microscopy, respectively. Analysis of tubby mutant worms with altered fat storage showed that, contrary to our hypothesis, increased adiposity did not sensitize to stressor-induced neurodegeneration. We further verify the accuracy of the algorithm by comparing code-generated, categorical degeneration results with manually scored dendrites of the same experiments. The platform, which detects 20 different metrics of neurodegeneration, can provide comparative insight into how each exposure affects dopaminergic neurodegeneration patterns.

RGC-Net: An Automatic Reconstruction and Quantification Algorithm for Retinal Ganglion Cells Based on Deep Learning

Purpose

The purpose of this study was to develop a deep learning-based fully automated reconstruction and quantification algorithm which automatically delineates the neurites and somas of retinal ganglion cells (RGCs).

Methods

We trained a deep learning-based multi-task image segmentation model, RGC-Net, that automatically segments the neurites and somas in RGC images. A total of 166 RGC scans with manual annotations from human experts were used to develop this model, whereas 132 scans were used for training, and the remaining 34 scans were reserved as testing data. Post-processing techniques removed speckles or dead cells in soma segmentation results to further improve the robustness of the model. Quantification analyses were also conducted to compare five different metrics obtained by our automated algorithm and manual annotations.

Results

Quantitatively, our segmentation model achieves average foreground accuracy, background accuracy, overall accuracy, and dice similarity coefficient of 0.692, 0.999, 0.997, and 0.691 for the neurite segmentation task, and 0.865, 0.999, 0.997, and 0.850 for the soma segmentation task, respectively.

Conclusions

The experimental results demonstrate that RGC-Net can accurately and reliably reconstruct neurites and somas in RGC images. We also demonstrate our algorithm is comparable to human manually curated annotations in quantification analyses.

Translational Relevance

Our deep learning model provides a new tool that can trace and analyze the RGC neurites and somas efficiently and faster than manual analysis.

NeuRegenerate: A Framework for Visualizing Neurodegeneration

Recent advances in high-resolution microscopy have allowed scientists to better understand the underlying brain connectivity. However, due to the limitation that biological specimens can only be imaged at a single timepoint, studying changes to neural projections over time is limited to observations gathered using population analysis. In this article, we introduce NeuRegenerate, a novel end-to-end framework for the prediction and visualization of changes in neural fiber morphology within a subject across specified age-timepoints. To predict projections, we present neuReGANerator, a deep-learning network based on cycle-consistent generative adversarial network (GAN) that translates features of neuronal structures across age-timepoints for large brain microscopy volumes. We improve the reconstruction quality of the predicted neuronal structures by implementing a density multiplier and a new loss function, called the hallucination loss. Moreover, to alleviate artifacts that occur due to tiling of large input volumes, we introduce a spatial-consistency module in the training pipeline of neuReGANerator. Finally, to visualize the change in projections, predicted using neuReGANerator, NeuRegenerate offers two modes: (i) neuroCompare to simultaneously visualize the difference in the structures of the neuronal projections, from two age domains (using structural view and bounded view), and (ii) neuroMorph, a vesselness-based morphing technique to interactively visualize the transformation of the structures from one age-timepoint to the other. Our framework is designed specifically for volumes acquired using wide-field microscopy. We demonstrate our framework by visualizing the structural changes within the cholinergic system of the mouse brain between a young and old specimen.

Unpaired mesh-to-image translation for 3D fluorescent microscopy images of neurons

While Generative Adversarial Networks (GANs) can now reliably produce realistic images in a multitude of imaging domains, they are ill-equipped to model thin, stochastic textures present in many large 3D fluorescent microscopy (FM) images acquired in biological research. This is especially problematic in neuroscience where the lack of ground truth data impedes the development of automated image analysis algorithms for neurons and neural populations. We therefore propose an unpaired mesh-to-image translation methodology for generating volumetric FM images of neurons from paired ground truths. We start by learning unique FM styles efficiently through a Gramian-based discriminator. Then, we stylize 3D voxelized meshes of previously reconstructed neurons by successively generating slices. As a result, we effectively create a synthetic microscope and can acquire realistic FM images of neurons with control over the image content and imaging configurations. We demonstrate the feasibility of our architecture and its superior performance compared to state-of-the-art image translation architectures through a variety of texture-based metrics, unsupervised segmentation accuracy, and an expert opinion test. In this study, we use 2 synthetic FM datasets and 2 newly acquired FM datasets of retinal neurons.

An unbiased, automated platform for scoring dopaminergic neurodegeneration in C. elegans

Caenorhabditis elegans ( C. elegans ) has served as a simple model organism to study dopaminergic neurodegeneration, as it enables quantitative analysis of cellular and sub-cellular morphologies in live animals. These isogenic nematodes have a rapid life cycle and transparent body, making high-throughput imaging and evaluation of fluorescently tagged neurons possible. However, the current state-of-the-art method for quantifying dopaminergic degeneration requires researchers to manually examine images and score dendrites into groups of varying levels of neurodegeneration severity, which is time consuming, subject to bias, and limited in data sensitivity. We aim to overcome the pitfalls of manual neuron scoring by developing an automated, unbiased image processing algorithm to quantify dopaminergic neurodegeneration in C. elegans . The algorithm can be used on images acquired with different microscopy setups and only requires two inputs: a maximum projection image of the four cephalic neurons in the C. elegans head and the pixel size of the user’s camera. We validate the platform by detecting and quantifying neurodegeneration in nematodes exposed to rotenone, cold shock, and 6-hydroxydopamine using 63x epifluorescence, 63x confocal, and 40x epifluorescence microscopy, respectively. Analysis of tubby mutant worms with altered fat storage showed that, contrary to our hypothesis, increased adiposity did not sensitize to stressor-induced neurodegeneration. We further verify the accuracy of the algorithm by comparing code-generated, categorical degeneration results with manually scored dendrites of the same experiments. The platform, which detects 19 different metrics of neurodegeneration, can provide comparative insight into how each exposure affects dopaminergic neurodegeneration patterns.

Tracing weak neuron fibers

MOTIVATION

Precise reconstruction of neuronal arbors is important for circuitry mapping. Many auto-tracing algorithms have been developed toward full reconstruction. However, it is still challenging to trace the weak signals of neurite fibers that often correspond to axons.

RESULTS

We proposed a method, named the NeuMiner, for tracing weak fibers by combining two strategies: an online sample mining strategy and a modified gamma transformation. NeuMiner improved the recall of weak signals (voxel values <20) by a large margin, from 5.1 to 27.8%. This is prominent for axons, which increased by 6.4 times, compared to 2.0 times for dendrites. Both strategies were shown to be beneficial for weak fiber recognition, and they reduced the average axonal spatial distances to gold standards by 46 and 13%, respectively. The improvement was observed on two prevalent automatic tracing algorithms and can be applied to any other tracers and image types.

AVAILABILITY AND IMPLEMENTATION

Source codes of NeuMiner are freely available on GitHub (https://github.com/crazylyf/neuronet/tree/semantic_fnm). Image visualization, preprocessing and tracing are conducted on the Vaa3D platform, which is accessible at the Vaa3D GitHub repository (https://github.com/Vaa3D). All training and testing images are cropped from high-resolution fMOST mouse brains downloaded from the Brain Image Library (https://www.brainimagelibrary.org/), and the corresponding gold standards are available at https://doi.brainimagelibrary.org/doi/10.35077/g.25.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Neuron tracing from light microscopy images: automation, deep learning and bench testing

MOTIVATION

Large-scale neuronal morphologies are essential to neuronal typing, connectivity characterization and brain modeling. It is widely accepted that automation is critical to the production of neuronal morphology. Despite previous survey papers about neuron tracing from light microscopy data in the last decade, thanks to the rapid development of the field, there is a need to update recent progress in a review focusing on new methods and remarkable applications.

RESULTS

This review outlines neuron tracing in various scenarios with the goal to help the community understand and navigate tools and resources. We describe the status, examples and accessibility of automatic neuron tracing. We survey recent advances of the increasingly popular deep-learning enhanced methods. We highlight the semi-automatic methods for single neuron tracing of mammalian whole brains as well as the resulting datasets, each containing thousands of full neuron morphologies. Finally, we exemplify the commonly used datasets and metrics for neuron tracing bench testing.

NeuroConstruct: 3D Reconstruction and Visualization of Neurites in Optical Microscopy Brain Images

We introduce NeuroConstruct, a novel end-to-end application for the segmentation, registration, and visualization of brain volumes imaged using wide-field microscopy. NeuroConstruct offers a Segmentation Toolbox with various annotation helper functions that aid experts to effectively and precisely annotate micrometer resolution neurites. It also offers an automatic neurites segmentation using convolutional neuronal networks (CNN) trained by the Toolbox annotations and somas segmentation using thresholding. To visualize neurites in a given volume, NeuroConstruct offers a hybrid rendering by combining iso-surface rendering of high-confidence classified neurites, along with real-time rendering of raw volume using a 2D transfer function for voxel classification score versus voxel intensity value. For a complete reconstruction of the 3D neurites, we introduce a Registration Toolbox that provides automatic coarse-to-fine alignment of serially sectioned samples. The quantitative and qualitative analysis show that NeuroConstruct outperforms the state-of-the-art in all design aspects. NeuroConstruct was developed as a collaboration between computer scientists and neuroscientists, with an application to the study of cholinergic neurons, which are severely affected in Alzheimer's disease.

Practical guide for preparation, computational reconstruction and analysis of 3D human neuronal networks in control and ischaemic conditions

To obtain commensurate numerical data of neuronal network morphology in vitro, network analysis needs to follow consistent guidelines. Important factors in successful analysis are sample uniformity, suitability of the analysis method for extracting relevant data and the use of established metrics. However, for the analysis of 3D neuronal cultures, there is little coherence in the analysis methods and metrics used in different studies. Here, we present a framework for the analysis of neuronal networks in 3D. First, we selected a hydrogel that supported the growth of human pluripotent stem cell-derived cortical neurons. Second, we tested and compared two software programs for tracing multi-neuron images in three dimensions and optimized a workflow for neuronal analysis using software that was considered highly suitable for this purpose. Third, as a proof of concept, we exposed 3D neuronal networks to oxygen-glucose deprivation- and ionomycin-induced damage and showed morphological differences between the damaged networks and control samples utilizing the proposed analysis workflow. With the optimized workflow, we present a protocol for preparing, challenging, imaging and analysing 3D human neuronal cultures.

Summary: An optimized protocol is presented that allows morphological, quantifiable differences between the damaged and control human neuronal networks to be detected in three-dimensional cultures.

Super-resolution Segmentation Network for Reconstruction of Packed Neurites

Neuron reconstruction can provide the quantitative data required for measuring the neuronal morphology and is crucial in brain research. However, the difficulty in reconstructing dense neurites, wherein massive labor is required for accurate reconstruction in most cases, has not been well resolved. In this work, we provide a new pathway for solving this challenge by proposing the super-resolution segmentation network (SRSNet), which builds the mapping of the neurites in the original neuronal images and their segmentation in a higher-resolution (HR) space. During the segmentation process, the distances between the boundaries of the packed neurites are enlarged, and only the central parts of the neurites are segmented. Owing to this strategy, the super-resolution segmented images are produced for subsequent reconstruction. We carried out experiments on neuronal images with a voxel size of 0.2 μm × 0.2 μm × 1 μm produced by fMOST. SRSNet achieves an average F1 score of 0.88 for automatic packed neurites reconstruction, which takes both the precision and recall values into account, while the average F1 scores of other state-of-the-art automatic tracing methods are less than 0.70.

Smart imaging to empower brain-wide neuroscience at single-cell levels

A deep understanding of the neuronal connectivity and networks with detailed cell typing across brain regions is necessary to unravel the mechanisms behind the emotional and memorial functions as well as to find the treatment of brain impairment. Brain-wide imaging with single-cell resolution provides unique advantages to access morphological features of a neuron and to investigate the connectivity of neuron networks, which has led to exciting discoveries over the past years based on animal models, such as rodents. Nonetheless, high-throughput systems are in urgent demand to support studies of neural morphologies at larger scale and more detailed level, as well as to enable research on non-human primates (NHP) and human brains. The advances in artificial intelligence (AI) and computational resources bring great opportunity to 'smart' imaging systems, i.e., to automate, speed up, optimize and upgrade the imaging systems with AI and computational strategies. In this light, we review the important computational techniques that can support smart systems in brain-wide imaging at single-cell resolution.

Bringing machine learning to research on intellectual and developmental disabilities: taking inspiration from neurological diseases

Intellectual and Developmental Disabilities (IDDs), such as Down syndrome, Fragile X syndrome, Rett syndrome, and autism spectrum disorder, usually manifest at birth or early childhood. IDDs are characterized by significant impairment in intellectual and adaptive functioning, and both genetic and environmental factors underpin IDD biology. Molecular and genetic stratification of IDDs remain challenging mainly due to overlapping factors and comorbidity. Advances in high throughput sequencing, imaging, and tools to record behavioral data at scale have greatly enhanced our understanding of the molecular, cellular, structural, and environmental basis of some IDDs. Fueled by the "big data" revolution, artificial intelligence (AI) and machine learning (ML) technologies have brought a whole new paradigm shift in computational biology. Evidently, the ML-driven approach to clinical diagnoses has the potential to augment classical methods that use symptoms and external observations, hoping to push the personalized treatment plan forward. Therefore, integrative analyses and applications of ML technology have a direct bearing on discoveries in IDDs. The application of ML to IDDs can potentially improve screening and early diagnosis, advance our understanding of the complexity of comorbidity, and accelerate the identification of biomarkers for clinical research and drug development. For more than five decades, the IDDRC network has supported a nexus of investigators at centers across the USA, all striving to understand the interplay between various factors underlying IDDs. In this review, we introduced fast-increasing multi-modal data types, highlighted example studies that employed ML technologies to illuminate factors and biological mechanisms underlying IDDs, as well as recent advances in ML technologies and their applications to IDDs and other neurological diseases. We discussed various molecular, clinical, and environmental data collection modes, including genetic, imaging, phenotypical, and behavioral data types, along with multiple repositories that store and share such data. Furthermore, we outlined some fundamental concepts of machine learning algorithms and presented our opinion on specific gaps that will need to be filled to accomplish, for example, reliable implementation of ML-based diagnosis technology in IDD clinics. We anticipate that this review will guide researchers to formulate AI and ML-based approaches to investigate IDDs and related conditions.

Deep-Learning-Based Automated Neuron Reconstruction From 3D Microscopy Images Using Synthetic Training Images

Digital reconstruction of neuronal structures from 3D microscopy images is critical for the quantitative investigation of brain circuits and functions. It is a challenging task that would greatly benefit from automatic neuron reconstruction methods. In this paper, we propose a novel method called SPE-DNR that combines spherical-patches extraction (SPE) and deep-learning for neuron reconstruction (DNR). Based on 2D Convolutional Neural Networks (CNNs) and the intensity distribution features extracted by SPE, it determines the tracing directions and classifies voxels into foreground or background. This way, starting from a set of seed points, it automatically traces the neurite centerlines and determines when to stop tracing. To avoid errors caused by imperfect manual reconstructions, we develop an image synthesizing scheme to generate synthetic training images with exact reconstructions. This scheme simulates 3D microscopy imaging conditions as well as structural defects, such as gaps and abrupt radii changes, to improve the visual realism of the synthetic images. To demonstrate the applicability and generalizability of SPE-DNR, we test it on 67 real 3D neuron microscopy images from three datasets. The experimental results show that the proposed SPE-DNR method is robust and competitive compared with other state-of-the-art neuron reconstruction methods.

Hidden Markov modeling for maximum probability neuron reconstruction

Recent advances in brain clearing and imaging have made it possible to image entire mammalian brains at sub-micron resolution. These images offer the potential to assemble brain-wide atlases of neuron morphology, but manual neuron reconstruction remains a bottleneck. Several automatic reconstruction algorithms exist, but most focus on single neuron images. In this paper, we present a probabilistic reconstruction method, ViterBrain, which combines a hidden Markov state process that encodes neuron geometry with a random field appearance model of neuron fluorescence. ViterBrain utilizes dynamic programming to compute the global maximizer of what we call the most probable neuron path. We applied our algorithm to imperfect image segmentations, and showed that it can follow axons in the presence of noise or nearby neurons. We also provide an interactive framework where users can trace neurons by fixing start and endpoints. ViterBrain is available in our open-source Python package brainlit.

ViterBrain is an automated probabilistic reconstruction method that can reconstruct neuronal geometry and processes from microscopy images with code available in the open-source Python package, brainlit.

Axonal and Dendritic Morphology of Excitatory Neurons in Layer 2/3 Mouse Barrel Cortex Imaged Through Whole-Brain Two-Photon Tomography and Registered to a Digital Brain Atlas

Communication between cortical areas contributes importantly to sensory perception and cognition. On the millisecond time scale, information is signaled from one brain area to another by action potentials propagating across long-range axonal arborizations. Here, we develop and test methodology for imaging and annotating the brain-wide axonal arborizations of individual excitatory layer 2/3 neurons in mouse barrel cortex through single-cell electroporation and two-photon serial section tomography followed by registration to a digital brain atlas. Each neuron had an extensive local axon within the barrel cortex. In addition, individual neurons innervated subsets of secondary somatosensory cortex; primary somatosensory cortex for upper limb, trunk, and lower limb; primary and secondary motor cortex; visual and auditory cortical regions; dorsolateral striatum; and various fiber bundles. In the future, it will be important to assess if the diversity of axonal projections across individual layer 2/3 mouse barrel cortex neurons is accompanied by functional differences in their activity patterns.

Seeing the Forest and Its Trees Together: Implementing 3D Light Microscopy Pipelines for Cell Type Mapping in the Mouse Brain

The brain is composed of diverse neuronal and non-neuronal cell types with complex regional connectivity patterns that create the anatomical infrastructure underlying cognition. Remarkable advances in neuroscience techniques enable labeling and imaging of these individual cell types and their interactions throughout intact mammalian brains at a cellular resolution allowing neuroscientists to examine microscopic details in macroscopic brain circuits. Nevertheless, implementing these tools is fraught with many technical and analytical challenges with a need for high-level data analysis. Here we review key technical considerations for implementing a brain mapping pipeline using the mouse brain as a primary model system. Specifically, we provide practical details for choosing methods including cell type specific labeling, sample preparation (e.g., tissue clearing), microscopy modalities, image processing, and data analysis (e.g., image registration to standard atlases). We also highlight the need to develop better 3D atlases with standardized anatomical labels and nomenclature across species and developmental time points to extend the mapping to other species including humans and to facilitate data sharing, confederation, and integrative analysis. In summary, this review provides key elements and currently available resources to consider while developing and implementing high-resolution mapping methods.

Classifying the tracing difficulty of 3D neuron image blocks based on deep learning

Quickly and accurately tracing neuronal morphologies in large-scale volumetric microscopy data is a very challenging task. Most automatic algorithms for tracing multi-neuron in a whole brain are designed under the Ultra-Tracer framework, which begins the tracing of a neuron from its soma and traces all signals via a block-by-block strategy. Some neuron image blocks are easy for tracing and their automatic reconstructions are very accurate, and some others are difficult and their automatic reconstructions are inaccurate or incomplete. The former are called low Tracing Difficulty Blocks (low-TDBs), while the latter are called high Tracing Difficulty Blocks (high-TDBs). We design a model named 3D-SSM to classify the tracing difficulty of 3D neuron image blocks, which is based on 3D Residual neural Network (3D-ResNet), Fully Connected Neural Network (FCNN) and Long Short-Term Memory network (LSTM). 3D-SSM contains three modules: Structure Feature Extraction (SFE), Sequence Information Extraction (SIE) and Model Fusion (MF). SFE utilizes a 3D-ResNet and a FCNN to extract two kinds of features in 3D image blocks and their corresponding automatic reconstruction blocks. SIE uses two LSTMs to learn sequence information hidden in 3D image blocks. MF adopts a concatenation operation and a FCNN to combine outputs from SIE. 3D-SSM can be used as a stop condition of an automatic tracing algorithm in the Ultra-Tracer framework. With its help, neuronal signals in low-TDBs can be traced by the automatic algorithm and in high-TDBs may be reconstructed by annotators. 12732 training samples and 5342 test samples are constructed on neuron images of a whole mouse brain. The 3D-SSM achieves classification accuracy rates 87.04% on the training set and 84.07% on the test set. Furthermore, the trained 3D-SSM is tested on samples from another whole mouse brain and its accuracy rate is 83.21%.

Weakly Supervised Neuron Reconstruction From Optical Microscopy Images With Morphological Priors

Manually labeling neurons from high-resolution but noisy and low-contrast optical microscopy (OM) images is tedious. As a result, the lack of annotated data poses a key challenge when applying deep learning techniques for reconstructing neurons from noisy and low-contrast OM images. While traditional tracing methods provide a possible way to efficiently generate labels for supervised network training, the generated pseudo-labels contain many noisy and incorrect labels, which lead to severe performance degradation. On the other hand, the publicly available dataset, BigNeuron, provides a large number of single 3D neurons that are reconstructed using various imaging paradigms and tracing methods. Though the raw OM images are not fully available for these neurons, they convey essential morphological priors for complex 3D neuron structures. In this paper, we propose a new approach to exploit morphological priors from neurons that have been reconstructed for training a deep neural network to extract neuron signals from OM images. We integrate a deep segmentation network in a generative adversarial network (GAN), expecting the segmentation network to be weakly supervised by pseudo-labels at the pixel level while utilizing the supervision of previously reconstructed neurons at the morphology level. In our morphological-prior-guided neuron reconstruction GAN, named MP-NRGAN, the segmentation network extracts neuron signals from raw images, and the discriminator network encourages the extracted neurons to follow the morphology distribution of reconstructed neurons. Comprehensive experiments on the public VISoR-40 dataset and BigNeuron dataset demonstrate that our proposed MP-NRGAN outperforms state-of-the-art approaches with less training effort.

Automated Neuron Tracing Using Content-Aware Adaptive Voxel Scooping on CNN Predicted Probability Map

Neuron tracing, as the essential step for neural circuit building and brain information flow analyzing, plays an important role in the understanding of brain organization and function. Though lots of methods have been proposed, automatic and accurate neuron tracing from optical images remains challenging. Current methods often had trouble in tracing the complex tree-like distorted structures and broken parts of neurite from a noisy background. To address these issues, we propose a method for accurate neuron tracing using content-aware adaptive voxel scooping on a convolutional neural network (CNN) predicted probability map. First, a 3D residual CNN was applied as preprocessing to predict the object probability and suppress high noise. Then, instead of tracing on the binary image produced by maximum classification, an adaptive voxel scooping method was presented for successive neurite tracing on the probability map, based on the internal content properties (distance, connectivity, and probability continuity along direction) of the neurite. Last, the neuron tree graph was built using the length first criterion. The proposed method was evaluated on the public BigNeuron datasets and fluorescence micro-optical sectioning tomography (fMOST) datasets and outperformed current state-of-art methods on images with neurites that had broken parts and complex structures. The high accuracy tracing proved the potential of the proposed method for neuron tracing on large-scale.

Neuron tracing and quantitative analyses of dendritic architecture reveal symmetrical three-way-junctions and phenotypes of git-1 in C. elegans

Complex dendritic trees are a distinctive feature of neurons. Alterations to dendritic morphology are associated with developmental, behavioral and neurodegenerative changes. The highly-arborized PVD neuron of C. elegans serves as a model to study dendritic patterning; however, quantitative, objective and automated analyses of PVD morphology are missing. Here, we present a method for neuronal feature extraction, based on deep-learning and fitting algorithms. The extracted neuronal architecture is represented by a database of structural elements for abstracted analysis. We obtain excellent automatic tracing of PVD trees and uncover that dendritic junctions are unevenly distributed. Surprisingly, these junctions are three-way-symmetrical on average, while dendritic processes are arranged orthogonally. We quantify the effect of mutation in git-1, a regulator of dendritic spine formation, on PVD morphology and discover a localized reduction in junctions. Our findings shed new light on PVD architecture, demonstrating the effectiveness of our objective analyses of dendritic morphology and suggest molecular control mechanisms.

Improving scalability in systems neuroscience

Emerging technologies to acquire data at increasingly greater scales promise to transform discovery in systems neuroscience. However, current exponential growth in the scale of data acquisition is a double-edged sword. Scaling up data acquisition can speed up the cycle of discovery but can also misinterpret the results or possibly slow down the cycle because of challenges presented by the curse of high-dimensional data. Active, adaptive, closed-loop experimental paradigms use hardware and algorithms optimized to enable time-critical computation to provide feedback that interprets the observations and tests hypotheses to actively update the stimulus or stimulation parameters. In this perspective, we review important concepts of active and adaptive experiments and discuss how selectively constraining the dimensionality and optimizing strategies at different stages of discovery loop can help mitigate the curse of high-dimensional data. Active and adaptive closed-loop experimental paradigms can speed up discovery despite an exponentially increasing data scale, offering a road map to timely and iterative hypothesis revision and discovery in an era of exponential growth in neuroscience.

Bridging the Brain and Data Sciences

Brain scientists are now capable of collecting more data in a single experiment than researchers a generation ago might have collected over an entire career. Indeed, the brain itself seems to thirst for more and more data. Such digital information not only comprises individual studies but is also increasingly shared and made openly available for secondary, confirmatory, and/or combined analyses. Numerous web resources now exist containing data across spatiotemporal scales. Data processing workflow technologies running via cloud-enabled computing infrastructures allow for large-scale processing. Such a move toward greater openness is fundamentally changing how brain science results are communicated and linked to available raw data and processed results. Ethical, professional, and motivational issues challenge the whole-scale commitment to data-driven neuroscience. Nevertheless, fueled by government investments into primary brain data collection coupled with increased sharing and community pressure challenging the dominant publishing model, large-scale brain and data science is here to stay.

Red blood cell phenotyping from 3D confocal images using artificial neural networks

The investigation of cell shapes mostly relies on the manual classification of 2D images, causing a subjective and time consuming evaluation based on a portion of the cell surface. We present a dual-stage neural network architecture for analyzing fine shape details from confocal microscopy recordings in 3D. The system, tested on red blood cells, uses training data from both healthy donors and patients with a congenital blood disease, namely hereditary spherocytosis. Characteristic shape features are revealed from the spherical harmonics spectrum of each cell and are automatically processed to create a reproducible and unbiased shape recognition and classification. The results show the relation between the particular genetic mutation causing the disease and the shape profile. With the obtained 3D phenotypes, we suggest our method for diagnostics and theragnostics of blood diseases. Besides the application employed in this study, our algorithms can be easily adapted for the 3D shape phenotyping of other cell types and extend their use to other applications, such as industrial automated 3D quality control.

Neuron Image Segmentation via Learning Deep Features and Enhancing Weak Neuronal Structures

Neuron morphology reconstruction (tracing) in 3D volumetric images is critical for neuronal research. However, most existing neuron tracing methods are not applicable in challenging datasets where the neuron images are contaminated by noises or containing weak filament signals. In this paper, we present a two-stage 3D neuron segmentation approach via learning deep features and enhancing weak neuronal structures, to reduce the impact of image noise in the data and enhance the weak-signal neuronal structures. In the first stage, we train a voxel-wise multi-level fully convolutional network (FCN), which specializes in learning deep features, to obtain the 3D neuron image segmentation maps in an end-to-end manner. In the second stage, a ray-shooting model is employed to detect the discontinued segments in segmentation results of the first-stage, and the local neuron diameter of the broken point is estimated and direction of the filamentary fragment is detected by rayburst sampling algorithm. Then, a Hessian-repair model is built to repair the broken structures, by enhancing weak neuronal structures in a fibrous structure determined by the estimated local neuron diameter and the filamentary fragment direction. Experimental results demonstrate that our proposed segmentation approach achieves better segmentation performance than other state-of-the-art methods for 3D neuron segmentation. Compared with the neuron reconstruction results on the segmented images produced by other segmentation methods, the proposed approach gains 47.83% and 34.83% improvement in the average distance scores. The average Precision and Recall rates of the branch point detection with our proposed method are 38.74% and 22.53% higher than the detection results without segmentation.

Neuron type classification in rat brain based on integrative convolutional and tree-based recurrent neural networks

The study of cellular complexity in the nervous system based on anatomy has shown more practical and objective advantages in morphology than other perspectives on molecular, physiological, and evolutionary aspects. However, morphology-based neuron type classification in the whole rat brain is challenging, given the significant number of neuron types, limited reconstructed neuron samples, and diverse data formats. Here, we report that different types of deep neural network modules may well process different kinds of features and that the integration of these submodules will show power on the representation and classification of neuron types. For SWC-format data, which are compressed but unstructured, we construct a tree-based recurrent neural network (Tree-RNN) module. For 2D or 3D slice-format data, which are structured but with large volumes of pixels, we construct a convolutional neural network (CNN) module. We also generate a virtually simulated dataset with two classes, reconstruct a CASIA rat-neuron dataset with 2.6 million neurons without labels, and select the NeuroMorpho-rat dataset with 35,000 neurons containing hierarchical labels. In the twelve-class classification task, the proposed model achieves state-of-the-art performance compared with other models, e.g., the CNN, RNN, and support vector machine based on hand-designed features.

Ballistocardiogram Artifact Reduction in Simultaneous EEG-fMRI Using Deep Learning

OBJECTIVE

The concurrent recording of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) is a technique that has received much attention due to its potential for combined high temporal and spatial resolution. However, the ballistocardiogram (BCG), a large-amplitude artifact caused by cardiac induced movement contaminates the EEG during EEG-fMRI recordings. Removal of BCG in software has generally made use of linear decompositions of the corrupted EEG. This is not ideal as the BCG signal propagates in a manner which is non-linearly dependent on the electrocardiogram (ECG). In this paper, we present a novel method for BCG artifact suppression using recurrent neural networks (RNNs).

METHODS

EEG signals were recovered by training RNNs on the nonlinear mappings between ECG and the BCG corrupted EEG. We evaluated our model's performance against the commonly used Optimal Basis Set (OBS) method at the level of individual subjects, and investigated generalization across subjects.

RESULTS

We show that our algorithm can generate larger average power reduction of the BCG at critical frequencies, while simultaneously improving task relevant EEG based classification.

CONCLUSION

The presented deep learning architecture can be used to reduce BCG related artifacts in EEG-fMRI recordings.

SIGNIFICANCE

We present a deep learning approach that can be used to suppress the BCG artifact in EEG-fMRI without the use of additional hardware. This method may have scope to be combined with current hardware methods, operate in real-time and be used for direct modeling of the BCG.

Neuronal Population Reconstruction From Ultra-Scale Optical Microscopy Images via Progressive Learning

Reconstruction of neuronal populations from ultra-scale optical microscopy (OM) images is essential to investigate neuronal circuits and brain mechanisms. The noises, low contrast, huge memory requirement, and high computational cost pose significant challenges in the neuronal population reconstruction. Recently, many studies have been conducted to extract neuron signals using deep neural networks (DNNs). However, training such DNNs usually relies on a huge amount of voxel-wise annotations in OM images, which are expensive in terms of both finance and labor. In this paper, we propose a novel framework for dense neuronal population reconstruction from ultra-scale images. To solve the problem of high cost in obtaining manual annotations for training DNNs, we propose a progressive learning scheme for neuronal population reconstruction (PLNPR) which does not require any manual annotations. Our PLNPR scheme consists of a traditional neuron tracing module and a deep segmentation network that mutually complement and progressively promote each other. To reconstruct dense neuronal populations from a terabyte-sized ultra-scale image, we introduce an automatic framework which adaptively traces neurons block by block and fuses fragmented neurites in overlapped regions continuously and smoothly. We build a dataset "VISoR-40" which consists of 40 large-scale OM image blocks from cortical regions of a mouse. Extensive experimental results on our VISoR-40 dataset and the public BigNeuron dataset demonstrate the effectiveness and superiority of our method on neuronal population reconstruction and single neuron reconstruction. Furthermore, we successfully apply our method to reconstruct dense neuronal populations from an ultra-scale mouse brain slice. The proposed adaptive block propagation and fusion strategies greatly improve the completeness of neurites in dense neuronal population reconstruction.

Skeleton optimization of neuronal morphology based on three-dimensional shape restrictions

Background

Neurons are the basic structural unit of the brain, and their morphology is a key determinant of their classification. The morphology of a neuronal circuit is a fundamental component in neuron modeling. Recently, single-neuron morphologies of the whole brain have been used in many studies. The correctness and completeness of semimanually traced neuronal morphology are credible. However, there are some inaccuracies in semimanual tracing results. The distance between consecutive nodes marked by humans is very long, spanning multiple voxels. On the other hand, the nodes are marked around the centerline of the neuronal fiber, not on the centerline. Although these inaccuracies do not seriously affect the projection patterns that these studies focus on, they reduce the accuracy of the traced neuronal skeletons. These small inaccuracies will introduce deviations into subsequent studies that are based on neuronal morphology files.

Results

We propose a neuronal digital skeleton optimization method to evaluate and make fine adjustments to a digital skeleton after neuron tracing. Provided that the neuronal fiber shape is smooth and continuous, we describe its physical properties according to two shape restrictions. One restriction is designed based on the grayscale image, and the other is designed based on geometry. These two restrictions are designed to finely adjust the digital skeleton points to the neuronal fiber centerline. With this method, we design the three-dimensional shape restriction workflow of neuronal skeleton adjustment computation. The performance of the proposed method has been quantitatively evaluated using synthetic and real neuronal image data. The results show that our method can reduce the difference between the traced neuronal skeleton and the centerline of the neuronal fiber. Furthermore, morphology metrics such as the neuronal fiber length and radius become more precise.

Conclusions

This method can improve the accuracy of a neuronal digital skeleton based on traced results. The greater the accuracy of the digital skeletons that are acquired, the more precise the neuronal morphologies that are analyzed will be.

A bird's-eye view of deep learning in bioimage analysis

Deep learning of artificial neural networks has become the de facto standard approach to solving data analysis problems in virtually all fields of science and engineering. Also in biology and medicine, deep learning technologies are fundamentally transforming how we acquire, process, analyze, and interpret data, with potentially far-reaching consequences for healthcare. In this mini-review, we take a bird's-eye view at the past, present, and future developments of deep learning, starting from science at large, to biomedical imaging, and bioimage analysis in particular.

Weakly Supervised Learning of 3D Deep Network for Neuron Reconstruction

Digital reconstruction or tracing of 3D tree-like neuronal structures from optical microscopy images is essential for understanding the functionality of neurons and reveal the connectivity of neuronal networks. Despite the existence of numerous tracing methods, reconstructing a neuron from highly noisy images remains challenging, particularly for neurites with low and inhomogeneous intensities. Conducting deep convolutional neural network (CNN)-based segmentation prior to neuron tracing facilitates an approach to solving this problem via separation of weak neurites from a noisy background. However, large manual annotations are needed in deep learning-based methods, which is labor-intensive and limits the algorithm's generalization for different datasets. In this study, we present a weakly supervised learning method of a deep CNN for neuron reconstruction without manual annotations. Specifically, we apply a 3D residual CNN as the architecture for discriminative neuronal feature extraction. We construct the initial pseudo-labels (without manual segmentation) of the neuronal images on the basis of an existing automatic tracing method. A weakly supervised learning framework is proposed via iterative training of the CNN model for improved prediction and refining of the pseudo-labels to update training samples. The pseudo-label was iteratively modified via mining and addition of weak neurites from the CNN predicted probability map on the basis of their tubularity and continuity. The proposed method was evaluated on several challenging images from the public BigNeuron and Diadem datasets, to fMOST datasets. Owing to the adaption of 3D deep CNNs and weakly supervised learning, the presented method demonstrates effective detection of weak neurites from noisy images and achieves results similar to those of the CNN model with manual annotations. The tracing performance was significantly improved by the proposed method on both small and large datasets (>100 GB). Moreover, the proposed method proved to be superior to several novel tracing methods on original images. The results obtained on various large-scale datasets demonstrated the generalization and high precision achieved by the proposed method for neuron reconstruction.

Joint Segmentation and Path Classification of Curvilinear Structures

Detection of curvilinear structures in images has long been of interest. One of the most challenging aspects of this problem is inferring the graph representation of the curvilinear network. Most existing delineation approaches first perform binary segmentation of the image and then refine it using either a set of hand-designed heuristics or a separate classifier that assigns likelihood to paths extracted from the pixel-wise prediction. In our work, we bridge the gap between segmentation and path classification by training a deep network that performs those two tasks simultaneously. We show that this approach is beneficial because it enforces consistency across the whole processing pipeline. We apply our approach on roads and neurons datasets.

Mapping mesoscale axonal projections in the mouse brain using a 3D convolutional network

The projection targets of a neuronal population are a key feature of its anatomical characteristics. Historically, tissue sectioning, confocal microscopy, and manual scoring of specific regions of interest have been used to generate coarse summaries of mesoscale projectomes. We present here TrailMap, a three-dimensional (3D) convolutional network for extracting axonal projections from intact cleared mouse brains imaged by light-sheet microscopy. TrailMap allows region-based quantification of total axon content in large and complex 3D structures after registration to a standard reference atlas. The identification of axonal structures as thin as one voxel benefits from data augmentation but also requires a loss function that tolerates errors in annotation. A network trained with volumes of serotonergic axons in all major brain regions can be generalized to map and quantify axons from thalamocortical, deep cerebellar, and cortical projection neurons, validating transfer learning as a tool to adapt the model to novel categories of axonal morphology. Speed of training, ease of use, and accuracy improve over existing tools without a need for specialized computing hardware. Given the recent emphasis on genetically and functionally defining cell types in neural circuit analysis, TrailMap will facilitate automated extraction and quantification of axons from these specific cell types at the scale of the entire mouse brain, an essential component of deciphering their connectivity.

3D Neuron Reconstruction in Tangled Neuronal Image With Deep Networks

Digital reconstruction or tracing of 3D neuron is essential for understanding the brain functions. While existing automatic tracing algorithms work well for the clean neuronal image with a single neuron, they are not robust to trace the neuron surrounded by nerve fibers. We propose a 3D U-Net-based network, namely 3D U-Net Plus, to segment the neuron from the surrounding fibers before the application of tracing algorithms. All the images in BigNeuron, the biggest available neuronal image dataset, contain clean neurons with no interference of nerve fibers, which are not practical to train the segmentation network. Based upon the BigNeuron images, we synthesize a SYNethic TAngled NEuronal Image dataset (SYNTANEI) to train the proposed network, by fusing the neurons with extracted nerve fibers. Due to the adoption of dropout, àtrous convolution and Àtrous Spatial Pyramid Pooling (ASPP), experimental results on the synthetic and real tangled neuronal images show that the proposed 3D U-Net Plus network achieved very promising segmentation results. The neurons reconstructed by the tracing algorithm using the segmentation result match significantly better with the ground truth than that using the original images.

Brain-Wide Shape Reconstruction of a Traced Neuron Using the Convex Image Segmentation Method

Neuronal shape reconstruction is a helpful technique for establishing neuron identity, inferring neuronal connections, mapping neuronal circuits, and so on. Advances in optical imaging techniques have enabled data collection that includes the shape of a neuron across the whole brain, considerably extending the scope of neuronal anatomy. However, such datasets often include many fuzzy neurites and many crossover regions that neurites are closely attached, which make neuronal shape reconstruction more challenging. In this study, we proposed a convex image segmentation model for neuronal shape reconstruction that segments a neurite into cross sections along its traced skeleton. Both the sparse nature of gradient images and the rule that fuzzy neurites usually have a small radius are utilized to improve neuronal shape reconstruction in regions with fuzzy neurites. Because the model is closely related to the traced skeleton point, we can use this relationship for identifying neurite with crossover regions. We demonstrated the performance of our model on various datasets, including those with fuzzy neurites and neurites with crossover regions, and we verified that our model could robustly reconstruct the neuron shape on a brain-wide scale.

TeraVR empowers precise reconstruction of complete 3-D neuronal morphology in the whole brain

Neuron morphology is recognized as a key determinant of cell type, yet the quantitative profiling of a mammalian neuron's complete three-dimensional (3-D) morphology remains arduous when the neuron has complex arborization and long projection. Whole-brain reconstruction of neuron morphology is even more challenging as it involves processing tens of teravoxels of imaging data. Validating such reconstructions is extremely laborious. We develop TeraVR, an open-source virtual reality annotation system, to address these challenges. TeraVR integrates immersive and collaborative 3-D visualization, interaction, and hierarchical streaming of teravoxel-scale images. Using TeraVR, we have produced precise 3-D full morphology of long-projecting neurons in whole mouse brains and developed a collaborative workflow for highly accurate neuronal reconstruction.

PAT-Probabilistic Axon Tracking for Densely Labeled Neurons in Large 3-D Micrographs

A major goal of contemporary neuroscience research is to map the structural connectivity of mammalian brain using microscopy imaging data. In this context, the reconstruction of densely labeled axons from two-photon microscopy images is a challenging and important task. The visually overlapping, crossing, and often strongly distorted images of the axons allow many ambiguous interpretations to be made. We address the problem of tracking axons in densely labeled samples of neurons in large image data sets acquired from marmoset brains. Our high-resolution images were acquired using two-photon microscopy and they provided whole brain coverage, occupying terabytes of memory. Both the image distortions and the large data set size frequently make it impractical to apply present-day neuron tracing algorithms to such data due to the optimization of such algorithms to the precise tracing of either single or sparse sets of neurons. Thus, new tracking techniques are needed. We propose a probabilistic axon tracking algorithm (PAT). PAT tackles the tracking of axons in two steps: locally (L-PAT) and globally (G-PAT). L-PAT is a probabilistic tracking algorithm that can tackle distorted, cluttered images of densely labeled axons. L-PAT divides a large micrograph into smaller image stacks. It then processes each image stack independently before mapping the axons in each image to a sparse model of axon trajectories. G-PAT merges the sparse L-PAT models into a single global model of axon trajectories by minimizing a global objective function using a probabilistic optimization method. We demonstrate the superior performance of PAT over standard approaches on synthetic data. Furthermore, we successfully apply PAT to densely labeled axons in large images acquired from marmoset brains.