Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites

Testing the reproducibility of ecological studies on insect behavior in a multi-laboratory setting identifies opportunities for improving experimental rigor

The reproducibility of studies involving insect species is an underexplored area in the broader discussion about poor reproducibility in science. Our study addresses this gap by conducting a systematic multi-laboratory investigation into the reproducibility of ecological studies on insect behavior. We implemented a 3 × 3 experimental design, incorporating three study sites, and three independent experiments on three insect species from different orders: the turnip sawfly (Athalia rosae, Hymenoptera), the meadow grasshopper (Pseudochorthippus parallelus, Orthoptera), and the red flour beetle (Tribolium castaneum, Coleoptera). Using random-effect meta-analysis, we compared the consistency and accuracy of treatment effects on insect behavioral traits across replicate experiments. We successfully reproduced the overall statistical treatment effect in 83% of the replicate experiments, but overall effect size replication was achieved in only 66% of the replicates. Thus, though demonstrating sufficient reproducibility in some measures, this study also provides the first experimental evidence for cases of poor reproducibility in insect experiments. Our findings further show that reasons causing poor reproducibility established in rodent research also hold for other study organisms and research questions. We believe that a rethinking of current best practices is required to face reproducibility issues in insect studies but also across disciplines. Specifically, we advocate for adopting open research practices and the implementation of methodological strategies that reduce bias and problems arising from over-standardization. With respect to the latter, the introduction of systematic variation through multi-laboratory or heterogenized designs may contribute to improved reproducibility in studies involving any living organisms.

Advances in cell-based biosensors: Transforming food flavor evaluation with novel approaches

Food flavor, a blend of taste and smell, is key to consumer acceptance and food quality. Traditional sensory and instrumental methods often fail to replicate human sensory responses. This review discusses the role of cell-based biosensors in flavor evaluation, showcasing their sensitivity, specificity, and rapid response. Using living cells like taste and olfactory cells, these biosensors surpass traditional approaches. Advancements include microelectrode array systems with taste receptor cells for real-time detection of bitter, sweet, and umami substances and improved cell immobilization technologies for detecting complex odorant profiles. Challenges such as signal stability, selective detection, cell cultivation, and scalability persist. However, integrating artificial intelligence and portable technologies could broaden their applications. With the potential to revolutionize sensory analysis, cell-based biosensors offer a sustainable, precise, and scalable approach to food flavor evaluation, bridging sensory perception with advanced analytical methods and driving innovation in food science.

A systematic assessment of robustness in CNS safety pharmacology

BACKGROUND AND PURPOSE

Irwin tests are key preclinical study elements for characterising drug-induced neurological side effects. This multicentre study aimed to assess the robustness of Irwin tests across multinational sites during three stages of protocol harmonisation. The projects were part of the Enhanced Quality in Preclinical Data framework, aiming to increase success rates in transition from preclinical testing to clinical application.

EXPERIMENTAL APPROACH

Female and male NMRI mice were assigned to one of three groups (vehicle, MK-801 0.1 and 0.3 mg kg-1). Irwin scores were assessed at baseline and multiple times following intraperitoneal injection of MK-801 using local protocols (Stage 1), shared protocols with harmonised environmental design (Stage 2) and fully harmonised Irwin scoring protocols (Stage 3).

KEY RESULTS

The analysis based on the four functional domains (motor, autonomic, sedation and excitation) revealed substantial data variability in Stages 1 and 2. Although there was still marked overall heterogeneity between sites in Stage 3 after complete harmonisation of the Irwin scoring scheme, heterogeneity was only moderate within functional domains. When comparing treatment groups versus vehicle, we found large effect sizes in the motor domain and subtle to moderate effects in the excitation-related and autonomic domains.

CONCLUSION AND IMPLICATIONS

The pronounced interlaboratory variability in Irwin datasets for the CNS-active compound MK-801 needs to be carefully considered when making decisions during drug development. While environmental and general study design had a minor impact, the study suggests that harmonisation of parameters and their scoring can limit variability and increase robustness.

A bespoke water T-maze apparatus and protocol: an optimized, reliable, and repeatable method for screening learning, memory, and executive functioning in laboratory mice

The Morris Water Maze (MWM) is the most commonly used assay for evaluating learning and memory in laboratory mice. Despite its widespread use, contemporary reviews have highlighted substantial methodological variation in experimental protocols and that the associated testing procedures are acutely (each trial) and chronically (testing across days) stressful; stress impairs attention, memory consolidation and the retrieval of learned information. Moreover, the interpretation of behavior within the MWM is often difficult because of wall hugging, non-spatial swim strategies, floating, and jumping off the escape platform. Together, these issues may compromise the reproducibility, generalizability, and predictability of experimental results, as well as animal welfare. To address these issues, and as an initial proof-of-principle, we first narrowed the spatial dimensions of the MWM by using a T-insert, which constrained and reduced the overall length of time/distance that the animal must swim in order to navigate to the escape platform, thus reducing stress and off-task behavior. Given the robust performance observed across spatial acquisition (learning and memory) as well as during reversal learning (executive function), we further reduced (by 43%) the overall distance and time that the animal must swim in order to find the escape platform in a bespoke standalone Water T-Maze (WTM). We show, across five experiments, procedural refinements to our protocol and demonstrate robust, reliable and reproducible indicators of learning, memory and executive functioning in a task that is also significantly more efficient (3 days of testing within the WTM vs. 11 days of testing within the MWM). Taken together, our WTM apparatus and protocol are a significant improvement over other water-based apparatuses and protocols for evaluating learning, memory, and executive functioning in laboratory mice.

Evaluation of variation in preclinical electroencephalographic (EEG) spectral power across multiple laboratories and experiments: An EQIPD study

The European Quality In Preclinical Data (EQIPD) consortium was born from the fact that publications report challenges with the robustness, rigor, and/or validity of research data, which may impact decisions about whether to proceed with further preclinical testing or to advance to clinical testing, as well as draw conclusions on the predictability of preclinical models. To address this, a consortium including multiple research laboratories from academia and industry participated in a series of electroencephalography (EEG) experiments in mice aimed to detect sources of variance and to gauge how protocol harmonisation and data analytics impact such variance. Ultimately, the goal of this first ever between-laboratory comparison of EEG recordings and analyses was to validate the principles that supposedly increase data quality, robustness, and comparability. Experiments consisted of a Localisation phase, which aimed to identify the factors that influence between-laboratory variability, a Harmonisation phase to evaluate whether harmonisation of standardized protocols and centralised processing and data analysis reduced variance, and a Ring-Testing phase to verify the ability of the harmonised protocol to generate consistent findings. Indeed, between-laboratory variability reduced from Localisation to Harmonisation and this reduction remained during the Ring-Testing phase. Results obtained in this multicentre preclinical qEEG study also confirmed the complex nature of EEG experiments starting from the surgery and data collection through data pre-processing to data analysis that ultimately influenced the results and contributed to variance in findings across laboratories. Overall, harmonisation of protocols and centralized data analysis were crucial in reducing laboratory-to-laboratory variability. To this end, it is recommended that standardized guidelines be updated and followed for collection and analysis of preclinical EEG data.

Mouse Exploratory Behaviour in the Open Field with and without NAT-1 EEG Device: Effects of MK801 and Scopolamine

One aspect of reproducibility in preclinical research that is frequently overlooked is the physical condition in which physiological, pharmacological, or behavioural recordings are conducted. In this study, the physical conditions of mice were altered through the attachments of wireless electrophysiological recording devices (Neural Activity Tracker-1, NAT-1). NAT-1 devices are miniaturised multichannel devices with onboard memory for direct high-resolution recording of brain activity for >48 h. Such devices may limit the mobility of animals and affect their behavioural performance due to the added weight (total weight of approximately 3.4 g). The mice were additionally treated with saline (control), N-methyl-D-aspartate (NMDA) receptor antagonist MK801 (0.85 mg/kg), or the muscarinic acetylcholine receptor blocker scopolamine (0.65 mg/kg) to allow exploration of the effect of NAT-1 attachments in pharmacologically treated mice. We found only minimal differences in behavioural outcomes with NAT-1 attachments in standard parameters of locomotor activity widely reported for the open field test between the drug treatments. Hypoactivity was globally observed as a consistent outcome in the MK801-treated mice and hyperactivity in scopolamine groups regardless of NAT-1 attachments. These data collectively confirm the reproducibility for combined behavioural, pharmacological, and physiological endpoints even in the presence of lightweight wireless data loggers. The NAT-1 therefore constitutes a pertinent tool for investigating brain activity in, e.g., drug discovery and models of neuropsychiatric and/or neurodegenerative diseases with minimal effects on pharmacological and behavioural outcomes.

Good Practice Guideline for Preclinical Alcohol Research: The STRINGENCY Framework

Research in the field of preclinical alcohol research, but also science in general, has a problem: Many published scientific results cannot be repeated. As a result, findings from preclinical research often do not translate well to humans, causing increasing disappointment and calls for restructuring of preclinical research, that is, better reproducibility of preclinical research. However, the replication crisis is an inherent problem in biomedical research. Replication failures are not only due to small experimental variations but are often the result of poor methodology. In response to the replication crisis, numerous guidelines and recommendations have been proposed to promote transparency, rigor, and reproducibility in scientific research. What is missing today is a framework that integrates all the confusing information that results from all these guidelines and recommendations. Here we present STRINGENCY, an integrative approach to good practice guidelines for preclinical alcohol research, which can also apply to behavioral research in general and which aims to improve preclinical research to better prepare it for translation and minimize the "valley of death" in translational research. STRINGENCY includes systematic review and, when possible, meta-analysis prior to study design, sample size calculation, preregistration, multisite experiments, scientific data management (FAIR), reporting of data using ARRIVE, generalization of research data, and transparent publications that allow reporting of null results. We invite the scientific community to adopt STRINGENCY to improve the reliability and impact of preclinical alcohol research.

Measuring the replicability of our own research

In the study of transgenic mouse models of neurodevelopmental and neurodegenerative disorders, we use batteries of tests to measure deficits in behaviour and from the results of these tests, we make inferences about the mental states of the mice that we interpret as deficits in "learning", "memory", "anxiety", "depression", etc. This paper discusses the problems of determining whether a particular transgenic mouse is a valid mouse model of disease X, the problem of background strains, and the question of whether our behavioural tests are measuring what we say they are. The problem of the reliability of results is then discussed: are they replicable between labs and can we replicate our results in our own lab? This involves the study of intra- and inter- experimenter reliability. The variables that influence replicability and the importance of conducting a complete behavioural phenotype: sensory, motor, cognitive and social emotional behaviour are discussed. Then the thorny question of failure to replicate is examined: Is it a curse or a blessing? Finally, the role of failure in research and what it tells us about our research paradigms is examined.

A systematic assessment of preclinical multilaboratory studies and a comparison to single laboratory studies

Background

Multicentric approaches are widely used in clinical trials to assess the generalizability of findings, however, they are novel in laboratory-based experimentation. It is unclear how multilaboratory studies may differ in conduct and results from single lab studies. Here, we synthesized the characteristics of these studies and quantitatively compared their outcomes to those generated by single laboratory studies.

Methods

MEDLINE and Embase were systematically searched. Screening and data extractions were completed in duplicate by independent reviewers. Multilaboratory studies investigating interventions using in vivo animal models were included. Study characteristics were extracted. Systematic searches were then performed to identify single lab studies matched by intervention and disease. Difference in standardized mean differences (DSMD) was then calculated across studies to assess differences in effect estimates based on study design (>0 indicates larger effects in single lab studies).

Results

Sixteen multilaboratory studies met inclusion criteria and were matched to 100 single lab studies. The multicenter study design was applied across a diverse range of diseases, including stroke, traumatic brain injury, myocardial infarction, and diabetes. The median number of centers was four (range 2-6) and the median sample size was 111 (range 23-384) with rodents most frequently used. Multilaboratory studies adhered to practices that reduce the risk of bias significantly more often than single lab studies. Multilaboratory studies also demonstrated significantly smaller effect sizes than single lab studies (DSMD 0.72 [95% confidence interval 0.43-1]).

Conclusions

Multilaboratory studies demonstrate trends that have been well recognized in clinical research (i.e. smaller treatment effects with multicentric evaluation and greater rigor in study design). This approach may provide a method to robustly assess interventions and the generalizability of findings between laboratories.

Funding

uOttawa Junior Clinical Research Chair; The Ottawa Hospital Anesthesia Alternate Funds Association; Canadian Anesthesia Research Foundation; Government of Ontario Queen Elizabeth II Graduate Scholarship in Science and Technology.