Reproducibility in Computational Neuroscience Models and Simulations

Guidance for sharing computational models of neural stimulation: from project planning to publication

Objective. Sharing computational models offers many benefits, including increased scientific rigor during project execution, readership of the associated paper, resource usage efficiency, replicability, and reusability. In recognition of the growing practice and requirement of sharing models, code, and data, herein, we provide guidance to facilitate sharing of computational models by providing an accessible resource for regular reference throughout a project's stages.Approach. We synthesized literature on good practices in scientific computing and on code and data sharing with our experience in developing, sharing, and using models of neural stimulation, although the guidance will also apply well to most other types of computational models.Main results. We first describe the '6 R' characteristics of shared models, leaning on prior scientific computing literature, which enforce accountability and enable advancement: re-runnability, repeatability, replicability, reproducibility, reusability, and readability. We then summarize action items associated with good practices in scientific computing, including selection of computational tools during project planning, code and documentation design during development, and user instructions for deployment. We provide a detailed checklist of the contents of shared models and associated materials, including the model itself, code for reproducing published figures, documentation, and supporting datasets. We describe code, model, and data repositories, including a list of characteristics to consider when selecting a platform for sharing. We describe intellectual property (IP) considerations to balance permissive, open-source licenses versus software patents and bespoke licenses that govern and incentivize commercialization. Finally, we exemplify these practices with our ASCENT pipeline for modeling peripheral nerve stimulation.Significance. We hope that this paper will serve as an important and actionable reference for scientists who develop models-from project planning through publication-as well as for model users, institutions, IP experts, journals, funding sources, and repository platform developers.

Easy-MODA: Simplifying standardised registration of scientific simulation workflows through MODA template guidelines powered by the Enalos Cloud Platform

Modelling Data (MODA) reporting guidelines have been proposed for common terminology and for recording metadata for physics-based materials modelling and simulations in a CEN Workshop Agreement (CWA 17284:2018). Their purpose is similar to that of the Quantitative Structure-Activity Relationship (QSAR) model report form (QMRF) that aims to increase industry and regulatory confidence in QSAR models, but for a wider range of model types. Recently, the WorldFAIR project's nanomaterials case study suggested that both QMRF and MODA templates are an important means to enhance compliance of nanoinformatics models, and their underpinning datasets, with the FAIR principles (Findable, Accessible, Interoperable, Reusable). Despite the advances in computational modelling of materials properties and phenomena, regulatory uptake of predictive models has been slow. This is, in part, due to concerns about lack of validation of complex models and lack of documentation of scientific simulations. The models are often complex, output can be hardware- and software-dependent, and there is a lack of shared standards. Despite advocating for standardised and transparent documentation of simulation protocols through its templates, the MODA guidelines are rarely used in practice by modellers because of a lack of tools for automating their creation, sharing, and storage. They also suffer from a paucity of user guidance on their use to document different types of models and systems. Such tools exist for the more well-established QMRF and have aided widespread implementation of QMRFs. To address this gap, a simplified procedure and online tool, Easy-MODA, has been developed to guide users through MODA creation for physics-based and data-based models, and their various combinations. Easy-MODA is available as a web-tool on the Enalos Cloud Platform (https://www.enaloscloud.novamechanics.com/insight/moda/). The tool streamlines the creation of detailed MODA documentation, even for complex multi-model workflows, and facilitates the registration of MODA workflows and documentation in a database, thereby increasing their Findability and thus Re-usability. This enhances communication, interoperability, and reproducibility in multiscale materials modelling and improves trust in the models through improved documentation. The use of the Easy-MODA tool is exemplified by a case study for nanotoxicity evaluation, involving interlinked models and data transformation, to demonstrate the effectiveness of the tool in integrating complex computational methodologies and its significant role in improving the FAIRness of scientific simulations.

Quasistatic approximation in neuromodulation

We define and explain the quasistatic approximation (QSA) as applied to field modeling for electrical and magnetic stimulation. Neuromodulation analysis pipelines include discrete stages, and QSA is applied specifically when calculating the electric and magnetic fields generated in tissues by a given stimulation dose. QSA simplifies the modeling equations to support tractable analysis, enhanced understanding, and computational efficiency. The application of QSA in neuromodulation is based on four underlying assumptions: (A1) no wave propagation or self-induction in tissue, (A2) linear tissue properties, (A3) purely resistive tissue, and (A4) non-dispersive tissue. As a consequence of these assumptions, each tissue is assigned a fixed conductivity, and the simplified equations (e.g. Laplace's equation) are solved for the spatial distribution of the field, which is separated from the field's temporal waveform. Recognizing that electrical tissue properties may be more complex, we explain how QSA can be embedded in parallel or iterative pipelines to model frequency dependence or nonlinearity of conductivity. We survey the history and validity of QSA across specific applications, such as microstimulation, deep brain stimulation, spinal cord stimulation, transcranial electrical stimulation, and transcranial magnetic stimulation. The precise definition and explanation of QSA in neuromodulation are essential for rigor when using QSA models or testing their limits.

Automating literature screening and curation with applications to computational neuroscience

OBJECTIVE

ModelDB (https://modeldb.science) is a discovery platform for computational neuroscience, containing over 1850 published model codes with standardized metadata. These codes were mainly supplied from unsolicited model author submissions, but this approach is inherently limited. For example, we estimate we have captured only around one-third of NEURON models, the most common type of models in ModelDB. To more completely characterize the state of computational neuroscience modeling work, we aim to identify works containing results derived from computational neuroscience approaches and their standardized associated metadata (eg, cell types, research topics).

MATERIALS AND METHODS

Known computational neuroscience work from ModelDB and identified neuroscience work queried from PubMed were included in our study. After pre-screening with SPECTER2 (a free document embedding method), GPT-3.5, and GPT-4 were used to identify likely computational neuroscience work and relevant metadata.

RESULTS

SPECTER2, GPT-4, and GPT-3.5 demonstrated varied but high abilities in identification of computational neuroscience work. GPT-4 achieved 96.9% accuracy and GPT-3.5 improved from 54.2% to 85.5% through instruction-tuning and Chain of Thought. GPT-4 also showed high potential in identifying relevant metadata annotations.

DISCUSSION

Accuracy in identification and extraction might further be improved by dealing with ambiguity of what are computational elements, including more information from papers (eg, Methods section), improving prompts, etc.

CONCLUSION

Natural language processing and large language model techniques can be added to ModelDB to facilitate further model discovery, and will contribute to a more standardized and comprehensive framework for establishing domain-specific resources.

Quasistatic approximation in neuromodulation

We define and explain the quasistatic approximation (QSA) as applied to field modeling for electrical and magnetic stimulation. Neuromodulation analysis pipelines include discrete stages, and QSA is applied specifically when calculating the electric and magnetic fields generated in tissues by a given stimulation dose. QSA simplifies the modeling equations to support tractable analysis, enhanced understanding, and computational efficiency. The application of QSA in neuro-modulation is based on four underlying assumptions: (A1) no wave propagation or self-induction in tissue, (A2) linear tissue properties, (A3) purely resistive tissue, and (A4) non-dispersive tissue. As a consequence of these assumptions, each tissue is assigned a fixed conductivity, and the simplified equations (e.g., Laplace's equation) are solved for the spatial distribution of the field, which is separated from the field's temporal waveform. Recognizing that electrical tissue properties may be more complex, we explain how QSA can be embedded in parallel or iterative pipelines to model frequency dependence or nonlinearity of conductivity. We survey the history and validity of QSA across specific applications, such as microstimulation, deep brain stimulation, spinal cord stimulation, transcranial electrical stimulation, and transcranial magnetic stimulation. The precise definition and explanation of QSA in neuromodulation are essential for rigor when using QSA models or testing their limits.

NRN-EZ: an application to streamline biophysical modeling of synaptic integration using NEURON

One of the fundamental goals in neuroscience is to determine how the brain processes information and ultimately controls the execution of complex behaviors. Over the past four decades, there has been a steady growth in our knowledge of the morphological and functional diversity of neurons, the building blocks of the brain. These cells clearly differ not only for their anatomy and ion channel distribution, but also for the type, strength, location, and temporal pattern of activity of the many synaptic inputs they receive. Compartmental modeling programs like NEURON have become widely used in the neuroscience community to address a broad range of research questions, including how neurons integrate synaptic inputs and propagate information through complex neural networks. One of the main strengths of NEURON is its ability to incorporate user-defined information about the realistic morphology and biophysical properties of different cell types. Although the graphical user interface of the program can be used to run initial exploratory simulations, introducing a stochastic representation of synaptic weights, locations and activation times typically requires users to develop their own codes, a task that can be overwhelming for some beginner users. Here we describe NRN-EZ, an interactive application that allows users to specify complex patterns of synaptic input activity that can be integrated as part of NEURON simulations. Through its graphical user interface, NRN-EZ aims to ease the learning curve to run computational models in NEURON, for users that do not necessarily have a computer science background.

Large-scale biophysically detailed model of somatosensory thalamocortical circuits in NetPyNE

The primary somatosensory cortex (S1) of mammals is critically important in the perception of touch and related sensorimotor behaviors. In 2015, the Blue Brain Project (BBP) developed a groundbreaking rat S1 microcircuit simulation with over 31,000 neurons with 207 morpho-electrical neuron types, and 37 million synapses, incorporating anatomical and physiological information from a wide range of experimental studies. We have implemented this highly detailed and complex S1 model in NetPyNE, using the data available in the Neocortical Microcircuit Collaboration Portal. NetPyNE provides a Python high-level interface to NEURON and allows defining complicated multiscale models using an intuitive declarative standardized language. It also facilitates running parallel simulations, automates the optimization and exploration of parameters using supercomputers, and provides a wide range of built-in analysis functions. This will make the S1 model more accessible and simpler to scale, modify and extend in order to explore research questions or interconnect to other existing models. Despite some implementation differences, the NetPyNE model preserved the original cell morphologies, electrophysiological responses and spatial distribution for all 207 cell types; and the connectivity properties of all 1941 pathways, including synaptic dynamics and short-term plasticity (STP). The NetPyNE S1 simulations produced reasonable physiological firing rates and activity patterns across all populations. When STP was included, the network generated a 1 Hz oscillation comparable to the original model in vitro-like state. By then reducing the extracellular calcium concentration, the model reproduced the original S1 in vivo-like states with asynchronous activity. These results validate the original study using a new modeling tool. Simulated local field potentials (LFPs) exhibited realistic oscillatory patterns and features, including distance- and frequency-dependent attenuation. The model was extended by adding thalamic circuits, including 6 distinct thalamic populations with intrathalamic, thalamocortical (TC) and corticothalamic connectivity derived from experimental data. The thalamic model reproduced single known cell and circuit-level dynamics, including burst and tonic firing modes and oscillatory patterns, providing a more realistic input to cortex and enabling study of TC interactions. Overall, our work provides a widely accessible, data-driven and biophysically-detailed model of the somatosensory TC circuits that can be employed as a community tool for researchers to study neural dynamics, function and disease.

Modernizing the NEURON Simulator for Sustainability, Portability, and Performance

The need for reproducible, credible, multiscale biological modeling has led to the development of standardized simulation platforms, such as the widely-used NEURON environment for computational neuroscience. Developing and maintaining NEURON over several decades has required attention to the competing needs of backwards compatibility, evolving computer architectures, the addition of new scales and physical processes, accessibility to new users, and efficiency and flexibility for specialists. In order to meet these challenges, we have now substantially modernized NEURON, providing continuous integration, an improved build system and release workflow, and better documentation. With the help of a new source-to-source compiler of the NMODL domain-specific language we have enhanced NEURON's ability to run efficiently, via the CoreNEURON simulation engine, on a variety of hardware platforms, including GPUs. Through the implementation of an optimized in-memory transfer mechanism this performance optimized backend is made easily accessible to users, providing training and model-development paths from laptop to workstation to supercomputer and cloud platform. Similarly, we have been able to accelerate NEURON's reaction-diffusion simulation performance through the use of just-in-time compilation. We show that these efforts have led to a growing developer base, a simpler and more robust software distribution, a wider range of supported computer architectures, a better integration of NEURON with other scientific workflows, and substantially improved performance for the simulation of biophysical and biochemical models.

A Modular Workflow for Performance Benchmarking of Neuronal Network Simulations

Modern computational neuroscience strives to develop complex network models to explain dynamics and function of brains in health and disease. This process goes hand in hand with advancements in the theory of neuronal networks and increasing availability of detailed anatomical data on brain connectivity. Large-scale models that study interactions between multiple brain areas with intricate connectivity and investigate phenomena on long time scales such as system-level learning require progress in simulation speed. The corresponding development of state-of-the-art simulation engines relies on information provided by benchmark simulations which assess the time-to-solution for scientifically relevant, complementary network models using various combinations of hardware and software revisions. However, maintaining comparability of benchmark results is difficult due to a lack of standardized specifications for measuring the scaling performance of simulators on high-performance computing (HPC) systems. Motivated by the challenging complexity of benchmarking, we define a generic workflow that decomposes the endeavor into unique segments consisting of separate modules. As a reference implementation for the conceptual workflow, we develop beNNch: an open-source software framework for the configuration, execution, and analysis of benchmarks for neuronal network simulations. The framework records benchmarking data and metadata in a unified way to foster reproducibility. For illustration, we measure the performance of various versions of the NEST simulator across network models with different levels of complexity on a contemporary HPC system, demonstrating how performance bottlenecks can be identified, ultimately guiding the development toward more efficient simulation technology.

Neuron-Glia Interactions and Brain Circuits

Recent evidence suggests that glial cells take an active role in a number of brain functions that were previously attributed solely to neurons. For example, astrocytes, one type of glial cells, have been shown to promote coordinated activation of neuronal networks, modulate sensory-evoked neuronal network activity, and influence brain state transitions during development. This reinforces the idea that astrocytes not only provide the "housekeeping" for the neurons, but that they also play a vital role in supporting and expanding the functions of brain circuits and networks. Despite this accumulated knowledge, the field of computational neuroscience has mostly focused on modeling neuronal functions, ignoring the glial cells and the interactions they have with the neurons. In this chapter, we introduce the biology of neuron-glia interactions, summarize the existing computational models and tools, and emphasize the glial properties that may be important in modeling brain functions in the future.

International data governance for neuroscience

As neuroscience projects increase in scale and cross international borders, different ethical principles, national and international laws, regulations, and policies for data sharing must be considered. These concerns are part of what is collectively called data governance. Whereas neuroscience data transcend borders, data governance is typically constrained within geopolitical boundaries. An international data governance framework and accompanying infrastructure can assist investigators, institutions, data repositories, and funders with navigating disparate policies. Here, we propose principles and operational considerations for how data governance in neuroscience can be navigated at an international scale and highlight gaps, challenges, and opportunities in a global brain data ecosystem. We consider how to approach data governance in a way that balances data protection requirements and the need for open science, so as to promote international collaboration through federated constructs such as the International Brain Initiative (IBI).

Evolution and Reproducibility of Simulation Modeling in Epidemiology and Health Policy Over Half a Century

Simulation models are increasingly being used to inform epidemiologic studies and health policy, yet there is great variation in their transparency and reproducibility. In this review, we provide an overview of applications of simulation models in health policy and epidemiology, analyze the use of best reporting practices, and assess the reproducibility of the models using predefined, categorical criteria. We identified and analyzed 1,613 applicable articles and found exponential growth in the number of studies over the past half century, with the highest growth in dynamic modeling approaches. The largest subset of studies focused on disease policy models (70%), within which pathological conditions, viral diseases, neoplasms, and cardiovascular diseases account for one-third of the articles. Model details were not reported in almost half of the studies. We also provide in-depth analysis of modeling best practices, reporting quality and reproducibility of models for a subset of 100 articles (50 highly cited and 50 randomly selected from the remaining articles). Only 7 of 26 in-depth evaluation criteria were satisfied by more than 80% of samples. We identify areas for increased application of simulation modeling and opportunities to enhance the rigor and documentation in the conduct and reporting of simulation modeling in epidemiology and health policy.

Touchscreen cognitive testing: Cross-species translation and co-clinical trials in neurodegenerative and neuropsychiatric disease

Translating results from pre-clinical animal studies to successful human clinical trials in neurodegenerative and neuropsychiatric disease presents a significant challenge. While this issue is clearly multifaceted, the lack of reproducibility and poor translational validity of many paradigms used to assess cognition in animal models are central contributors to this challenge. Computer-automated cognitive test batteries have the potential to substantially improve translation between pre-clinical studies and clinical trials by increasing both reproducibility and translational validity. Given the structured nature of data output, computer-automated tests also lend themselves to increased data sharing and other open science good practices. Over the past two decades, computer automated, touchscreen-based cognitive testing methods have been developed for non-human primate and rodent models. These automated methods lend themselves to increased standardization, hence reproducibility, and have become increasingly important for the elucidation of the neurobiological basis of cognition in animal models. More recently, there have been increased efforts to use these methods to enhance translational validity by developing task batteries that are nearly identical across different species via forward (i.e., translating animal tasks to humans) and reverse (i.e., translating human tasks to animals) translation. An additional benefit of the touchscreen approach is that a cross-species cognitive test battery makes it possible to implement co-clinical trials-an approach developed initially in cancer research-for novel treatments for neurodegenerative disorders. Co-clinical trials bring together pre-clinical and early clinical studies, which facilitates testing of novel treatments in mouse models with underlying genetic or other changes, and can help to stratify patients on the basis of genetic, molecular, or cognitive criteria. This approach can help to determine which patients should be enrolled in specific clinical trials and can facilitate repositioning and/or repurposing of previously approved drugs. This has the potential to mitigate the resources required to study treatment responses in large numbers of human patients.

CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility

The traditional scientific paper falls short of effectively communicating computational research.  To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.

CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility

The traditional scientific paper falls short of effectively communicating computational research.  To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.

Systematic Parameter Reviews in Cognitive Modeling: Towards a Robust and Cumulative Characterization of Psychological Processes in the Diffusion Decision Model

Parametric cognitive models are increasingly popular tools for analyzing data obtained from psychological experiments. One of the main goals of such models is to formalize psychological theories using parameters that represent distinct psychological processes. We argue that systematic quantitative reviews of parameter estimates can make an important contribution to robust and cumulative cognitive modeling. Parameter reviews can benefit model development and model assessment by providing valuable information about the expected parameter space, and can facilitate the more efficient design of experiments. Importantly, parameter reviews provide crucial-if not indispensable-information for the specification of informative prior distributions in Bayesian cognitive modeling. From the Bayesian perspective, prior distributions are an integral part of a model, reflecting cumulative theoretical knowledge about plausible values of the model's parameters (Lee, 2018). In this paper we illustrate how systematic parameter reviews can be implemented to generate informed prior distributions for the Diffusion Decision Model (DDM; Ratcliff and McKoon, 2008), the most widely used model of speeded decision making. We surveyed the published literature on empirical applications of the DDM, extracted the reported parameter estimates, and synthesized this information in the form of prior distributions. Our parameter review establishes a comprehensive reference resource for plausible DDM parameter values in various experimental paradigms that can guide future applications of the model. Based on the challenges we faced during the parameter review, we formulate a set of general and DDM-specific suggestions aiming to increase reproducibility and the information gained from the review process.

Non-replicability circumstances in a neural network model with Hodgkin-Huxley-type neurons

Building upon previous experiments can be used to accomplish new goals. In computing, it is imperative to reuse computer code to continue development on specific projects. Reproducibility is a fundamental building block in science, and experimental reproducibility issues have recently been of great concern. It may be surprising that reproducibility is also of concern in computational science. In this study, we used a previously published code to investigate neural network activity and we were unable to replicate our original results. This led us to investigate the code in question, and we found that several different aspects, attributable to floating-point arithmetic, were the cause of these replicability issues. Furthermore, we uncovered other manifestations of this lack of replicability in other parts of the computation with this model. The simulated model is a standard system of ordinary differential equations, very much like those commonly used in computational neuroscience. Thus, we believe that other researchers in the field should be vigilant when using such models and avoid drawing conclusions from calculations if their qualitative results can be substantially modified through non-reproducible circumstances.

Enabling Large-Scale Simulations With the GENESIS Neuronal Simulator

In this paper, we evaluate the computational performance of the GEneral NEural SImulation System (GENESIS) for large scale simulations of neural networks. While many benchmark studies have been performed for large scale simulations with leaky integrate-and-fire neurons or neuronal models with only a few compartments, this work focuses on higher fidelity neuronal models represented by 50–74 compartments per neuron. After making some modifications to the source code for GENESIS and its parallel implementation, PGENESIS, particularly to improve memory usage, we find that PGENESIS is able to efficiently scale on supercomputing resources to network sizes as large as 9 × 10⁶ neurons with 18 × 10⁹ synapses and 2.2 × 10⁶ neurons with 45 × 10⁹ synapses. The modifications to GENESIS that enabled these large scale simulations have been incorporated into the May 2019 Official Release of PGENESIS 2.4 available for download from the GENESIS web site (genesis-sim.org).

Transparency in Health Economic Modeling: Options, Issues and Potential Solutions

Economic models are increasingly being used by health economists to assess the value of health technologies and inform healthcare decision making. However, most published economic models represent a kind of black box, with known inputs and outputs but undisclosed internal calculations and assumptions. This lack of transparency makes the evaluation of the model results challenging, complicates comparisons between models, and limits the reproducibility of the models. Here, we aim to provide an overview of the possible steps that could be undertaken to make economic models more transparent and encourage model developers to share more detailed calculations and assumptions with their peers. Scenarios with different levels of transparency (i.e., how much information is disclosed) and reach of transparency (i.e., who has access to the disclosed information) are discussed, and five key concerns (copyrights, model misuse, confidential data, software, and time/resources) pertaining to model transparency are presented, along with possible solutions. While a shift toward open-source models is underway in health economics, as has happened before in other research fields, the challenges ahead should not be underestimated. Importantly, there is a pressing need to find an acceptable trade-off between the added value of model transparency and the time and resources needed to achieve such transparency. To this end, it will be crucial to set incentives at different stakeholder levels. Despite the many challenges, the many benefits of publicly sharing economic models make increased transparency a goal worth pursuing.

Transparency in Health Economic Modeling: Options, Issues and Potential Solutions

Economic models are increasingly being used by health economists to assess the value of health technologies and inform healthcare decision making. However, most published economic models represent a kind of black box, with known inputs and outputs but undisclosed internal calculations and assumptions. This lack of transparency makes the evaluation of the model results challenging, complicates comparisons between models, and limits the reproducibility of the models. Here, we aim to provide an overview of the possible steps that could be undertaken to make economic models more transparent and encourage model developers to share more detailed calculations and assumptions with their peers. Scenarios with different levels of transparency (i.e., how much information is disclosed) and reach of transparency (i.e., who has access to the disclosed information) are discussed, and five key concerns (copyrights, model misuse, confidential data, software, and time/resources) pertaining to model transparency are presented, along with possible solutions. While a shift toward open-source models is underway in health economics, as has happened before in other research fields, the challenges ahead should not be underestimated. Importantly, there is a pressing need to find an acceptable trade-off between the added value of model transparency and the time and resources needed to achieve such transparency. To this end, it will be crucial to set incentives at different stakeholder levels. Despite the many challenges, the many benefits of publicly sharing economic models make increased transparency a goal worth pursuing.

NetPyNE, a tool for data-driven multiscale modeling of brain circuits

Biophysical modeling of neuronal networks helps to integrate and interpret rapidly growing and disparate experimental datasets at multiple scales. The NetPyNE tool (www.netpyne.org) provides both programmatic and graphical interfaces to develop data-driven multiscale network models in NEURON. NetPyNE clearly separates model parameters from implementation code. Users provide specifications at a high level via a standardized declarative language, for example connectivity rules, to create millions of cell-to-cell connections. NetPyNE then enables users to generate the NEURON network, run efficiently parallelized simulations, optimize and explore network parameters through automated batch runs, and use built-in functions for visualization and analysis - connectivity matrices, voltage traces, spike raster plots, local field potentials, and information theoretic measures. NetPyNE also facilitates model sharing by exporting and importing standardized formats (NeuroML and SONATA). NetPyNE is already being used to teach computational neuroscience students and by modelers to investigate brain regions and phenomena.

A demonstration of modularity, reuse, reproducibility, portability and scalability for modeling and simulation of cardiac electrophysiology using Kepler Workflows

Multi-scale computational modeling is a major branch of computational biology as evidenced by the US federal interagency Multi-Scale Modeling Consortium and major international projects. It invariably involves specific and detailed sequences of data analysis and simulation, often with multiple tools and datasets, and the community recognizes improved modularity, reuse, reproducibility, portability and scalability as critical unmet needs in this area. Scientific workflows are a well-recognized strategy for addressing these needs in scientific computing. While there are good examples if the use of scientific workflows in bioinformatics, medical informatics, biomedical imaging and data analysis, there are fewer examples in multi-scale computational modeling in general and cardiac electrophysiology in particular. Cardiac electrophysiology simulation is a mature area of multi-scale computational biology that serves as an excellent use case for developing and testing new scientific workflows. In this article, we develop, describe and test a computational workflow that serves as a proof of concept of a platform for the robust integration and implementation of a reusable and reproducible multi-scale cardiac cell and tissue model that is expandable, modular and portable. The workflow described leverages Python and Kepler-Python actor for plotting and pre/post-processing. During all stages of the workflow design, we rely on freely available open-source tools, to make our workflow freely usable by scientists.

We present a computational workflow as a proof of concept for integration and implementation of a reusable and reproducible cardiac multi-scale electrophysiology model that is expandable, modular and portable. This framework enables scientists to create intuitive, user-friendly and flexible end-to-end automated scientific workflows using a graphical user interface. Kepler is an advanced open-source platform that supports multiple models of computation. The underlying workflow engine handles scalability, provenance, reproducibility aspects of the code, performs orchestration of data flow, and automates execution on heterogeneous computing resources. One of the main advantages of workflow utilization is the integration of code written in multiple languages Standardization occurs at the interfaces of the workflow elements and allows for general applications and easy comparison and integration of code from different research groups or even multiple programmers coding in different languages for various purposes from the same group. A workflow driven problem-solving approach enables domain scientists to focus on resolving the core science questions, and delegates the computational and process management burden to the underlying Workflow. The workflow driven approach allows scaling the computational experiment with distributed data-parallel execution on multiple computing platforms, such as, HPC resources, GPU clusters, Cloud etc. The workflow framework tracks software version information along with hardware information to allow users an opportunity to trace any variation in workflow outcome to the system configurations.

Recent advances in biomedical simulations: a manifesto for model engineering

Biomedical simulations are widely used to understand disease, engineer cells, and model cellular processes. In this article, we explore how to improve the quality of biomedical simulations by developing simulation models using tools and practices employed in software engineering. We refer to this direction as model engineering. Not all techniques used by software engineers are directly applicable to model engineering, and so some adaptations are required. That said, we believe that simulation models can benefit from software engineering practices for requirements, design, and construction as well as from software engineering tools for version control, error checking, and testing. Here we survey current efforts to improve simulation quality and discuss promising research directions for model engineering.

Reproducible Neural Network Simulations: Statistical Methods for Model Validation on the Level of Network Activity Data

Computational neuroscience relies on simulations of neural network models to bridge the gap between the theory of neural networks and the experimentally observed activity dynamics in the brain. The rigorous validation of simulation results against reference data is thus an indispensable part of any simulation workflow. Moreover, the availability of different simulation environments and levels of model description require also validation of model implementations against each other to evaluate their equivalence. Despite rapid advances in the formalized description of models, data, and analysis workflows, there is no accepted consensus regarding the terminology and practical implementation of validation workflows in the context of neural simulations. This situation prevents the generic, unbiased comparison between published models, which is a key element of enhancing reproducibility of computational research in neuroscience. In this study, we argue for the establishment of standardized statistical test metrics that enable the quantitative validation of network models on the level of the population dynamics. Despite the importance of validating the elementary components of a simulation, such as single cell dynamics, building networks from validated building blocks does not entail the validity of the simulation on the network scale. Therefore, we introduce a corresponding set of validation tests and present an example workflow that practically demonstrates the iterative model validation of a spiking neural network model against its reproduction on the SpiNNaker neuromorphic hardware system. We formally implement the workflow using a generic Python library that we introduce for validation tests on neural network activity data. Together with the companion study (Trensch et al., 2018), the work presents a consistent definition, formalization, and implementation of the verification and validation process for neural network simulations.

A Systematic Review of Continuum Modeling of Skeletal Muscles: Current Trends, Limitations, and Recommendations

Finite elasticity theory has been commonly used to model skeletal muscle. A very large range of heterogeneous constitutive laws has been proposed. In this review, the most widely used continuum models of skeletal muscles were synthetized and discussed. Trends and limitations of these laws were highlighted to propose new recommendations for future researches. A systematic review process was performed using two reliable search engines as PubMed and ScienceDirect. 40 representative studies (13 passive muscle materials and 27 active muscle materials) were included into this review. Note that exclusion criteria include tendon models, analytical models, 1D geometrical models, supplement papers, and indexed conference papers. Trends of current skeletal muscle modeling relate to 3D accurate muscle representation, parameter identification in passive muscle modeling, and the integration of coupled biophysical phenomena. Parameter identification for active materials, assumed fiber distribution, data assumption, and model validation are current drawbacks. New recommendations deal with the incorporation of multimodal data derived from medical imaging, the integration of more biophysical phenomena, and model reproducibility. Accounting for data uncertainty in skeletal muscle modeling will be also a challenging issue. This review provides, for the first time, a holistic view of current continuum models of skeletal muscles to identify potential gaps of current models according to the physiology of skeletal muscle. This opens new avenues for improving skeletal muscle modeling in the framework of in silico medicine.

Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail

Replicability and reproducibility of computational models has been somewhat understudied by "the replication movement." In this paper, we draw on methodological studies into the replicability of psychological experiments and on the mechanistic account of explanation to analyze the functions of model replications and model reproductions in computational neuroscience. We contend that model replicability, or independent researchers' ability to obtain the same output using original code and data, and model reproducibility, or independent researchers' ability to recreate a model without original code, serve different functions and fail for different reasons. This means that measures designed to improve model replicability may not enhance (and, in some cases, may actually damage) model reproducibility. We claim that although both are undesirable, low model reproducibility poses more of a threat to long-term scientific progress than low model replicability. In our opinion, low model reproducibility stems mostly from authors' omitting to provide crucial information in scientific papers and we stress that sharing all computer code and data is not a solution. Reports of computational studies should remain selective and include all and only relevant bits of code.

Tellurium notebooks-An environment for reproducible dynamical modeling in systems biology

The considerable difficulty encountered in reproducing the results of published dynamical models limits validation, exploration and reuse of this increasingly large biomedical research resource. To address this problem, we have developed Tellurium Notebook, a software system for model authoring, simulation, and teaching that facilitates building reproducible dynamical models and reusing models by 1) providing a notebook environment which allows models, Python code, and narrative to be intermixed, 2) supporting the COMBINE archive format during model development for capturing model information in an exchangeable format and 3) enabling users to easily simulate and edit public COMBINE-compliant models from public repositories to facilitate studying model dynamics, variants and test cases. Tellurium Notebook, a Python–based Jupyter–like environment, is designed to seamlessly inter-operate with these community standards by automating conversion between COMBINE standards formulations and corresponding in–line, human–readable representations. Thus, Tellurium brings to systems biology the strategy used by other literate notebook systems such as Mathematica. These capabilities allow users to edit every aspect of the standards–compliant models and simulations, run the simulations in–line, and re–export to standard formats. We provide several use cases illustrating the advantages of our approach and how it allows development and reuse of models without requiring technical knowledge of standards. Adoption of Tellurium should accelerate model development, reproducibility and reuse.

There is considerable value to systems and synthetic biology in creating reproducible models. An essential element of reproducibility is the use of community standards, an often challenging undertaking for modelers. This article describes Tellurium Notebook, a tool for developing dynamical models that provides an intuitive approach to building and reusing models built with community standards. Tellurium automates embedding human–readable representations of COMBINE archives in literate coding notebooks, bringing to systems biology this strategy central to other literate notebook systems such as Mathematica. We show that the ability to easily edit this human–readable representation enables users to test models under a variety of conditions, thereby providing a way to create, reuse, and modify standard–encoded models and simulations, regardless of the user’s level of technical knowledge of said standards.

Challenges in Reproducibility, Replicability, and Comparability of Computational Models and Tools for Neuronal and Glial Networks, Cells, and Subcellular Structures

The possibility to replicate and reproduce published research results is one of the biggest challenges in all areas of science. In computational neuroscience, there are thousands of models available. However, it is rarely possible to reimplement the models based on the information in the original publication, let alone rerun the models just because the model implementations have not been made publicly available. We evaluate and discuss the comparability of a versatile choice of simulation tools: tools for biochemical reactions and spiking neuronal networks, and relatively new tools for growth in cell cultures. The replicability and reproducibility issues are considered for computational models that are equally diverse, including the models for intracellular signal transduction of neurons and glial cells, in addition to single glial cells, neuron-glia interactions, and selected examples of spiking neuronal networks. We also address the comparability of the simulation results with one another to comprehend if the studied models can be used to answer similar research questions. In addition to presenting the challenges in reproducibility and replicability of published results in computational neuroscience, we highlight the need for developing recommendations and good practices for publishing simulation tools and computational models. Model validation and flexible model description must be an integral part of the tool used to simulate and develop computational models. Constant improvement on experimental techniques and recording protocols leads to increasing knowledge about the biophysical mechanisms in neural systems. This poses new challenges for computational neuroscience: extended or completely new computational methods and models may be required. Careful evaluation and categorization of the existing models and tools provide a foundation for these future needs, for constructing multiscale models or extending the models to incorporate additional or more detailed biophysical mechanisms. Improving the quality of publications in computational neuroscience, enabling progressive building of advanced computational models and tools, can be achieved only through adopting publishing standards which underline replicability and reproducibility of research results.

The Spectrum of Mechanism-Oriented Models and Methods for Explanations of Biological Phenomena

Developing and improving mechanism-oriented computational models to better explain biological phenomena is a dynamic and expanding frontier. As the complexity of targeted phenomena has increased, so too has the diversity in methods and terminologies, often at the expense of clarity, which can make reproduction challenging, even problematic. To encourage improved semantic and methodological clarity, we describe the spectrum of Mechanism-oriented Models being used to develop explanations of biological phenomena. We cluster explanations of phenomena into three broad groups. We then expand them into seven workflow-related model types having distinguishable features. We name each type and illustrate with examples drawn from the literature. These model types may contribute to the foundation of an ontology of mechanism-based biomedical simulation research. We show that the different model types manifest and exert their scientific usefulness by enhancing and extending different forms and degrees of explanation. The process starts with knowledge about the phenomenon and continues with explanatory and mathematical descriptions. Those descriptions are transformed into software and used to perform experimental explorations by running and examining simulation output. The credibility of inferences is thus linked to having easy access to the scientific and technical provenance from each workflow stage.

Credibility, Replicability, and Reproducibility in Simulation for Biomedicine and Clinical Applications in Neuroscience

Modeling and simulation in computational neuroscience is currently a research enterprise to better understand neural systems. It is not yet directly applicable to the problems of patients with brain disease. To be used for clinical applications, there must not only be considerable progress in the field but also a concerted effort to use best practices in order to demonstrate model credibility to regulatory bodies, to clinics and hospitals, to doctors, and to patients. In doing this for neuroscience, we can learn lessons from long-standing practices in other areas of simulation (aircraft, computer chips), from software engineering, and from other biomedical disciplines. In this manuscript, we introduce some basic concepts that will be important in the development of credible clinical neuroscience models: reproducibility and replicability; verification and validation; model configuration; and procedures and processes for credible mechanistic multiscale modeling. We also discuss how garnering strong community involvement can promote model credibility. Finally, in addition to direct usage with patients, we note the potential for simulation usage in the area of Simulation-Based Medical Education, an area which to date has been primarily reliant on physical models (mannequins) and scenario-based simulations rather than on numerical simulations.

Computational Models for Calcium-Mediated Astrocyte Functions

The computational neuroscience field has heavily concentrated on the modeling of neuronal functions, largely ignoring other brain cells, including one type of glial cell, the astrocytes. Despite the short history of modeling astrocytic functions, we were delighted about the hundreds of models developed so far to study the role of astrocytes, most often in calcium dynamics, synchronization, information transfer, and plasticity in vitro, but also in vascular events, hyperexcitability, and homeostasis. Our goal here is to present the state-of-the-art in computational modeling of astrocytes in order to facilitate better understanding of the functions and dynamics of astrocytes in the brain. Due to the large number of models, we concentrated on a hundred models that include biophysical descriptions for calcium signaling and dynamics in astrocytes. We categorized the models into four groups: single astrocyte models, astrocyte network models, neuron-astrocyte synapse models, and neuron-astrocyte network models to ease their use in future modeling projects. We characterized the models based on which earlier models were used for building the models and which type of biological entities were described in the astrocyte models. Features of the models were compared and contrasted so that similarities and differences were more readily apparent. We discovered that most of the models were basically generated from a small set of previously published models with small variations. However, neither citations to all the previous models with similar core structure nor explanations of what was built on top of the previous models were provided, which made it possible, in some cases, to have the same models published several times without an explicit intention to make new predictions about the roles of astrocytes in brain functions. Furthermore, only a few of the models are available online which makes it difficult to reproduce the simulation results and further develop the models. Thus, we would like to emphasize that only via reproducible research are we able to build better computational models for astrocytes, which truly advance science. Our study is the first to characterize in detail the biophysical and biochemical mechanisms that have been modeled for astrocytes.

Physiological models of the lateral superior olive

In computational biology, modeling is a fundamental tool for formulating, analyzing and predicting complex phenomena. Most neuron models, however, are designed to reproduce certain small sets of empirical data. Hence their outcome is usually not compatible or comparable with other models or datasets, making it unclear how widely applicable such models are. In this study, we investigate these aspects of modeling, namely credibility and generalizability, with a specific focus on auditory neurons involved in the localization of sound sources. The primary cues for binaural sound localization are comprised of interaural time and level differences (ITD/ILD), which are the timing and intensity differences of the sound waves arriving at the two ears. The lateral superior olive (LSO) in the auditory brainstem is one of the locations where such acoustic information is first computed. An LSO neuron receives temporally structured excitatory and inhibitory synaptic inputs that are driven by ipsi- and contralateral sound stimuli, respectively, and changes its spike rate according to binaural acoustic differences. Here we examine seven contemporary models of LSO neurons with different levels of biophysical complexity, from predominantly functional ones (‘shot-noise’ models) to those with more detailed physiological components (variations of integrate-and-fire and Hodgkin-Huxley-type). These models, calibrated to reproduce known monaural and binaural characteristics of LSO, generate largely similar results to each other in simulating ITD and ILD coding. Our comparisons of physiological detail, computational efficiency, predictive performances, and further expandability of the models demonstrate (1) that the simplistic, functional LSO models are suitable for applications where low computational costs and mathematical transparency are needed, (2) that more complex models with detailed membrane potential dynamics are necessary for simulation studies where sub-neuronal nonlinear processes play important roles, and (3) that, for general purposes, intermediate models might be a reasonable compromise between simplicity and biological plausibility.

Computational models help our understanding of complex biological systems, by identifying their key elements and revealing their operational principles. Close comparisons between model predictions and empirical observations ensure our confidence in a model as a building block for further applications. Most current neuronal models, however, are constructed to replicate only a small specific set of experimental data. Thus, it is usually unclear how these models can be generalized to different datasets and how they compare with each other. In this paper, seven neuronal models are examined that are designed to reproduce known physiological characteristics of auditory neurons involved in the detection of sound source location. Despite their different levels of complexity, the models generate largely similar results when their parameters are tuned with common criteria. Comparisons show that simple models are computationally more efficient and theoretically transparent, and therefore suitable for rigorous mathematical analyses and engineering applications including real-time simulations. In contrast, complex models are necessary for investigating the relationship between underlying biophysical processes and sub- and suprathreshold spiking properties, although they have a large number of unconstrained, unverified parameters. Having identified their advantages and drawbacks, these auditory neuron models may readily be used for future studies and applications.

Multiscale modeling in the clinic: diseases of the brain and nervous system

Computational neuroscience is a field that traces its origins to the efforts of Hodgkin and Huxley, who pioneered quantitative analysis of electrical activity in the nervous system. While also continuing as an independent field, computational neuroscience has combined with computational systems biology, and neural multiscale modeling arose as one offshoot. This consolidation has added electrical, graphical, dynamical system, learning theory, artificial intelligence and neural network viewpoints with the microscale of cellular biology (neuronal and glial), mesoscales of vascular, immunological and neuronal networks, on up to macroscales of cognition and behavior. The complexity of linkages that produces pathophysiology in neurological, neurosurgical and psychiatric disease will require multiscale modeling to provide understanding that exceeds what is possible with statistical analysis or highly simplified models: how to bring together pharmacotherapeutics with neurostimulation, how to personalize therapies, how to combine novel therapies with neurorehabilitation, how to interlace periodic diagnostic updates with frequent reevaluation of therapy, how to understand a physical disease that manifests as a disease of the mind. Multiscale modeling will also help to extend the usefulness of animal models of human diseases in neuroscience, where the disconnects between clinical and animal phenomenology are particularly pronounced. Here we cover areas of particular interest for clinical application of these new modeling neurotechnologies, including epilepsy, traumatic brain injury, ischemic disease, neurorehabilitation, drug addiction, schizophrenia and neurostimulation.

Twenty years of ModelDB and beyond: building essential modeling tools for the future of neuroscience

Neuron modeling may be said to have originated with the Hodgkin and Huxley action potential model in 1952 and Rall's models of integrative activity of dendrites in 1964. Over the ensuing decades, these approaches have led to a massive development of increasingly accurate and complex data-based models of neurons and neuronal circuits. ModelDB was founded in 1996 to support this new field and enhance the scientific credibility and utility of computational neuroscience models by providing a convenient venue for sharing them. It has grown to include over 1100 published models covering more than 130 research topics. It is actively curated and developed to help researchers discover and understand models of interest. ModelDB also provides mechanisms to assist running models both locally and remotely, and has a graphical tool that enables users to explore the anatomical and biophysical properties that are represented in a model. Each of its capabilities is undergoing continued refinement and improvement in response to user experience. Large research groups (Allen Brain Institute, EU Human Brain Project, etc.) are emerging that collect data across multiple scales and integrate that data into many complex models, presenting new challenges of scale. We end by predicting a future for neuroscience increasingly fueled by new technology and high performance computation, and increasingly in need of comprehensive user-friendly databases such as ModelDB to provide the means to integrate the data for deeper insights into brain function in health and disease.

Simulation Neurotechnologies for Advancing Brain Research: Parallelizing Large Networks in NEURON

Large multiscale neuronal network simulations are of increasing value as more big data are gathered about brain wiring and organization under the auspices of a current major research initiative, such as Brain Research through Advancing Innovative Neurotechnologies. The development of these models requires new simulation technologies. We describe here the current use of the NEURON simulator with message passing interface (MPI) for simulation in the domain of moderately large networks on commonly available high-performance computers (HPCs). We discuss the basic layout of such simulations, including the methods of simulation setup, the run-time spike-passing paradigm, and postsimulation data storage and data management approaches. Using the Neuroscience Gateway, a portal for computational neuroscience that provides access to large HPCs, we benchmark simulations of neuronal networks of different sizes (500-100,000 cells), and using different numbers of nodes (1-256). We compare three types of networks, composed of either Izhikevich integrate-and-fire neurons (I&F), single-compartment Hodgkin-Huxley (HH) cells, or a hybrid network with half of each. Results show simulation run time increased approximately linearly with network size and decreased almost linearly with the number of nodes. Networks with I&F neurons were faster than HH networks, although differences were small since all tested cells were point neurons with a single compartment.