Improving preclinical studies through replications

Utilizing the Pronated Forearm Technique for Measuring Glenohumeral External Rotation in Baseball Players

BACKGROUND

Alterations in glenohumeral internal rotation (GIR), glenohumeral external rotation (GER), and total range of motion (TROM) have been linked with increased injury risk. GER capacity has been measured routinely with the forearm in neutral rotation (GERN), but a recent study reported GERN was greater than GER with the forearm in pronation (GERP) in Minor League pitchers. This work has not yet been replicated or extended to other groups.

HYPOTHESIS

GERP would be significantly less than GERN in Independent League baseball pitchers, and there would be no difference in GERP or GERN measurements between this new group and the previous group of Minor League pitchers.

STUDY DESIGN

Cross-sectional study.

LEVEL OF EVIDENCE

Level 3.

METHODS

Goniometric measurements were recorded for bilateral GIR, GERN, and GERP, and resulting TROM for 37 Independent League baseball pitchers. These data were compared with the previous study. All motions were compared individually between groups, between throwing and nonthrowing arm, and both within and between techniques (forearm neutral or pronated).

RESULTS

GERP was significantly less than GERN for both arms within each group tested (P < 0.01). Independent League pitchers had greater between arm differences for GIR (-16.9° vs -6.9°), GERN (+15.1° vs -0.6°), and GERP (+13.1° vs -5.9°) compared with Minor League pitchers. TROM for the Independent League pitchers was not statistically different for either measurement technique, while TROM for the throwing arm of the Minor League pitchers was statistically reduced with varying effect sizes (d = 0.35-0.99) compared with the nonthrowing arm (P < 0.01).

CONCLUSION

This study confirmed earlier findings that the pronated forearm resulted in decreased GER capacity, illustrating the adaptive response to throwing and the need to evaluate for this variable.

CLINICAL RELEVANCE

GERP should be evaluated in all groups of pitchers, but there may be variations within tested groups.

Preclinical systematic review of CCR5 antagonists as cerebroprotective and stroke recovery enhancing agents

C-C chemokine receptor type 5 (CCR5) antagonists may improve both acute stroke outcome and long-term recovery. Despite their evaluation in ongoing clinical trials, gaps remain in the evidence supporting their use. With a panel of patients with lived experiences of stroke, we performed a systematic review of animal models of stroke that administered a CCR5 antagonist and assessed infarct size or behavioural outcomes. MEDLINE, Web of Science, and Embase were searched. Article screening and data extraction were completed in duplicate. We pooled outcomes using random effects meta-analyses. We assessed risk of bias using the Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE) tool and alignment with the Stroke Treatment Academic Industry Roundtable (STAIR) and Stroke Recovery and Rehabilitation Roundtable (SRRR) recommendations. Five studies representing 10 experiments were included. CCR5 antagonists reduced infarct volume (standard mean difference -1.02; 95% confidence interval -1.58 to -0.46) when compared to stroke-only controls. Varied timing of CCR5 administration (pre- or post-stroke induction) produced similar benefit. CCR5 antagonists significantly improved 11 of 16 behavioural outcomes reported. High risk of bias was present in all studies and critical knowledge gaps in the preclinical evidence were identified using STAIR/SRRR. CCR5 antagonists demonstrate promise; however, rigorously designed preclinical studies that better align with STAIR/SRRR recommendations and downstream clinical trials are warranted. Prospective Register of Systematic Reviews (PROSPERO CRD42023393438).

A three-stage strategy for conducting an experimental investigation: A recommendation to improve the reproducibility of reported conclusions

The reproducibility of the results from preclinical research rests on many factors, including the selection of appropriate experimental designs for the individual experiments that constitute the investigation. The design of each of these experiments depends on their purpose within the entire investigation and the information to be gained from conducting them. Here, we explain and justify a three-stage strategy comprising a series of different types of experiment, each with a different purpose and design: a pilot study, a hypothesis-generating experiment and a final hypothesis-confirming experiment. Compliance with this three-stage strategy, over the course of an entire investigation, will not only strengthen its reproducibility but, importantly, can save time and other resources, including the total number of animals used.

Subdivided Historical Data to Assess Replicability of the Rat Embryo-Fetal Developmental Toxicity Study

INTRODUCTION

A key aspect of scientific reliability includes replicability, that is, obtaining consistent results when an experiment is repeated. In embryo-fetal developmental toxicity (EFDT) studies, replicability can be assessed using in vitro models, targeted in vivo studies, and/or the second species study. This work assesses the replicability of whole-animal studies using historic rat data.

METHODS

Data for two endpoints from five full studies were downloaded from the National Toxicology Program (NTP) website. Each full group was divided into two replicate sets (based on odd/even and top/bottom animal order) to evaluate within-study replicability. Analyses included summary statistics, scatter plots, a modified Levene's test for homogeneity of variances, and Cohen's d to assess effect sizes.

RESULTS

Replicate means deviated from the original study by only 0.4%-3.7% and differed by ≤ 7% between replicates (with differences < 5% in 87% of groups). Coefficients of variation (CV%) were generally consistent across subgroups, with few above 10%. Variance testing revealed significant differences in two of the five studies, and one study exhibited opposite fetal weight effects in the odd/even subgroup only. Evaluations of adjusted maternal weight gain were comparable across subgroups.

CONCLUSIONS

The observed 5%-7% differences between these idealized replicates may represent the lower bound for acceptable variability when merging replicate data sets. This work lays the groundwork for more robust evaluations of replicability in EFDT studies and may inform future regulatory guidance.

Laboratory and physiological aspects of substitute metazoan models for in vivo pharmacotoxicological analysis

New methods are essential to characterize the performance of substitute procedures for detecting therapeutic action(s) of a chemical or key signal of toxicological events. Herein, it was discussed the applications and advantages of using arthropods, worms, and fishes in pharmacological and/or toxicology assessments. First of all, the illusion of similarity covers many differences between humans and mice, remarkably about liver injury and metabolism of xenobiotics. Using invertebrates, especially earthworms (Eisenia fetida), brine shrimps (Artemia salina, Daphnia magna), and insects (Drosophila melanogaster) and vertebrates as small fishes (Oryzias latipes, Pimephales promelas, Danio rerio) has countless advantages, including fewer ethical conflicts, short life cycle, high reproduction rate, simpler to handle, and less complex anatomy. They can be used to find contaminants in organic matters and water and are easier genetically engineered with orthologous-mutated genes to explore specific proteins involved in proliferative and hormonal disturbances, chemotherapy multidrug resistance, and carcinogenicity. As multicellular embryos, larvae, and mature organisms, they can be tested in bigger-sized replication platforms with 24-, 96-, or 384-multiwell plates as cheaper and faster ways to select hit compounds from drug-like libraries to predict acute, subacute or chronic toxicity, pharmacokinetics, and efficacy parameters of pharmaceutical, cosmetic, and personal care products. Meanwhile, sublethal exposures are designed to identify changes in reproduction, body weight, DNA damages, oxidation, and immune defense responses in earthworms and zebrafishes, and swimming behaviors in A. salina and D. rerio. Behavioral parameters also give specificities on sublethal effects that would not be detected in zebrafishes by OECD protocols.

Emerging biophysical techniques for probing synaptic transmission in neurodegenerative disorders

Plethora of research has shed light on the critical role of synaptic dysfunction in various neurodegenerative disorders (NDDs), including Alzheimer's disease (AD), Parkinson's disease (PD), Amyotrophic lateral sclerosis (ALS), and Huntington's disease (HD). Synapses, the fundamental units for neural communication in the brain, are highly vulnerable to pathological conditions and are central to the progression of neurological diseases. The presynaptic terminal, a key component of synapses responsible for neurotransmitter release and synaptic communication, undergoes structural and functional alterations in these disorders. Understanding synaptic transmission abnormalities is crucial for unravelling the pathophysiological mechanisms underlying neurodegeneration. In the quest to probe synaptic transmission in NDDs, emerging biophysical techniques play a pivotal role. These advanced methods offer insights into the structural and functional changes occurring at nerve terminals in conditions like AD, PD, HD & ALS. By investigating synaptic plasticity and alterations in neurotransmitter release dynamics, researchers can uncover valuable information about disease progression and potential therapeutic targets. The review articles highlighted provide a comprehensive overview of how synaptic vulnerability and pathology are shared mechanisms across a spectrum of neurological disorders. In major neurodegenerative diseases, synaptic dysfunction is a common thread linking these conditions. The intricate molecular machinery involved in neurotransmitter release, synaptic vesicle dynamics, and presynaptic protein regulation are key areas of focus for understanding synaptic alterations in neurodegenerative diseases.

Standardized Protocol for Resazurin-Based Viability Assays on A549 Cell Line for Improving Cytotoxicity Data Reliability

The A549 cell line has become a cornerstone in biomedical research, particularly in cancer studies and serves as a critical tool in cytotoxicity studies and drug screening where it is used to evaluate the impact of pharmaceutical compounds on cellular viability. One of the most widely adopted methods for viability assessment, which is also used in evaluating drug cytotoxicity, is the resazurin-based assay. This assay exploits the ability of living cells to convert resazurin into fluorescent resorufin, providing a reliable indicator of metabolic activity. By measuring this conversion, cell viability can be estimated. Resazurin assay is extensively used for evaluating cytotoxic effects on various cell lines, including A549 cells, thereby bridging the gap between in vitro experimentation and drug development. However, frequent data inconsistencies in pre-clinical drug screening highlight the critical need for standardization to ensure reliability and reproducibility. This manuscript addresses these challenges by describing the optimization of resazurin-based viability assays for A549 cells in both 2D cultures and 3D fibrin gel models. By optimizing this test, the study aims to enhance the reliability of cytotoxicity results and introduces a new standard operating procedure, thus providing consistent results with minimal measurement uncertainty. This standardization is crucial for advancing drug screening and ensuring robust research findings.

Reproducible lung protective effects of a TGFβR1/ALK5 inhibitor in a bleomycin-induced and spirometry-confirmed model of IPF in male mice

This study comprehensively validated the bleomycin (BLEO) induced mouse model of IPF for utility in preclinical drug discovery. To this end, the model was rigorously evaluated for reproducible phenotype and TGFβ-directed treatment outcomes. Lung disease was profiled longitudinally in male C57BL6/JRJ mice receiving a single intratracheal instillation of BLEO (n = 10-12 per group). A TGFβR1/ALK5 inhibitor (ALK5i) was profiled in six independent studies in BLEO-IPF mice, randomized/stratified to treatment according to baseline body weight and non-invasive whole-body plethysmography. ALK5i (60 mg/kg/day) or vehicle (n = 10-16 per study) was administered orally for 21 days, starting 7 days after intratracheal BLEO installation. BLEO-IPF mice recapitulated functional, histological and biochemical hallmarks of IPF, including declining expiratory/inspiratory capacity and inflammatory and fibrotic lung injury accompanied by markedly elevated TGFβ levels in bronchoalveolar lavage fluid and lung tissue. Pulmonary transcriptome signatures of inflammation and fibrosis in BLEO-IPF mice were comparable to reported data in IPF patients. ALK5i promoted reproducible and robust therapeutic outcomes on lung functional, biochemical and histological endpoints in BLEO-IPF mice. The robust lung fibrotic disease phenotype, along with the consistent and reproducible lung protective effects of ALK5i treatment, makes the spirometry-confirmed BLEO-IPF mouse model highly applicable for profiling novel drug candidates for IPF.

Stroke of Consistency: Streamlining Multicenter Protocols for Enhanced Reproducibility of Infarct Volumes in Preclinical Stroke Research

BACKGROUND

The discrepancy between experimental research and clinical trial outcomes is a persistent challenge in preclinical studies, particularly in stroke research. A possible factor contributing to this issue is the lack of standardization across experimental stroke models, leading to poor reproducibility in multicenter studies. This study addresses this gap by aiming to enhance reproducibility and the efficacy of multicenter studies through the harmonization of protocols and training of involved personnel.

METHODS

We established a set of standard operating procedures for various stroke models and the Neuroscore. These standard operating procedures were implemented across multiple research centers, followed by specialized, in-person training for all participants. We measured the variability in infarct volume both before and after the implementation of these standardized protocols and training sessions.

RESULTS

The standardization process led to a significant reduction in variability of infarct volume across different stroke models (40%-50% reduction), demonstrating the effectiveness of our harmonized protocols and training. Additionally, the implementation of the Neuroscore system across centers showed low variability and consistent results up to 28 days poststroke, underscoring its utility in chronic phase evaluations.

CONCLUSIONS

The harmonization of protocols and surgeon training significantly reduced variability in experimental outcomes across different centers. This improvement can increase the comparability of data between research groups and enhance the statistical power of multicenter studies. Our findings also establish the Neuroscore as a reliable tool for long-term assessment in stroke research, paving the way for more consistent and impactful multicenter preclinical studies.

Good Practice Guideline for Preclinical Alcohol Research: The STRINGENCY Framework

Research in the field of preclinical alcohol research, but also science in general, has a problem: Many published scientific results cannot be repeated. As a result, findings from preclinical research often do not translate well to humans, causing increasing disappointment and calls for restructuring of preclinical research, that is, better reproducibility of preclinical research. However, the replication crisis is an inherent problem in biomedical research. Replication failures are not only due to small experimental variations but are often the result of poor methodology. In response to the replication crisis, numerous guidelines and recommendations have been proposed to promote transparency, rigor, and reproducibility in scientific research. What is missing today is a framework that integrates all the confusing information that results from all these guidelines and recommendations. Here we present STRINGENCY, an integrative approach to good practice guidelines for preclinical alcohol research, which can also apply to behavioral research in general and which aims to improve preclinical research to better prepare it for translation and minimize the "valley of death" in translational research. STRINGENCY includes systematic review and, when possible, meta-analysis prior to study design, sample size calculation, preregistration, multisite experiments, scientific data management (FAIR), reporting of data using ARRIVE, generalization of research data, and transparent publications that allow reporting of null results. We invite the scientific community to adopt STRINGENCY to improve the reliability and impact of preclinical alcohol research.

Growth capacity of a Wharton's Jelly derived mesenchymal stromal cells tissue engineered vascular graft used for main pulmonary artery reconstruction in piglets

Background: Surgical treatment of congenital heart defects affecting the right ventricular outflow tract (RVOT) often requires complex reconstruction and multiple reoperations due to structural degeneration and lack of growth of currently available materials. Hence, alternative approaches for RVOT reconstruction, which meet the requirements of biocompatibility and long-term durability of an ideal scaffold, are needed. Through this full scale pre-clinical study, we demonstrated the growth capacity of a Wharton's Jelly derived mesenchymal stromal cells (WJ-MSC) tissue engineered vascular graft used in reconstructing the main pulmonary artery in piglets, providing proof of biocompatibility and efficacy. Methods: Sixteen four-week-old Landrace pigs were randomized to undergo supravalvar Main Pulmonary Artery (MPA) replacement with either unseeded or WJ-MSCs-seeded Small Intestinal Submucosa-derived grafts. Animals were followed up for 6 months by clinical examinations and cardiac imaging. At termination, sections of MPAs were assessed by macroscopic inspection, histology and fluorescent immunohistochemistry. Results: Data collected at 6 months follow up showed no sign of graft thrombosis or calcification. The explanted main pulmonary arteries demonstrated a significantly higher degree of cellular organization and elastin content in the WJ-MSCs seeded grafts compared to the acellular counterparts. Transthoracic echocardiography and cardiovascular magnetic resonance confirmed the superior growth and remodelling of the WJ-MSCs seeded conduit compared to the unseeded. Conclusion: Our findings indicate that the addition of WJ-MSCs to the acellular scaffold can upgrade the material, converting it into a biologically active tissue, with the potential to grow, repair and remodel the RVOT.

Reducing the false discovery rate of preclinical animal research with Bayesian statistical decision criteria

The success of preclinical research hinges on exploratory and confirmatory animal studies. Traditional null hypothesis significance testing is a common approach to eliminate the chaff from a collection of drugs, so that only the most promising treatments are funneled through to clinical research phases. Balancing the number of false discoveries and false omissions is an important aspect to consider during this process. In this paper, we compare several preclinical research pipelines, either based on null hypothesis significance testing or based on Bayesian statistical decision criteria. We build on a recently published large-scale meta-analysis of reported effect sizes in preclinical animal research and elicit a non-informative prior distribution under which both approaches are compared. After correcting for publication bias and shrinkage of effect sizes in replication studies, simulations show that (i) a shift towards statistical approaches which explicitly incorporate the minimum clinically important difference reduces the false discovery rate of frequentist approaches and (ii) a shift towards Bayesian statistical decision criteria can improve the reliability of preclinical animal research by reducing the number of false-positive findings. It is shown that these benefits hold while keeping the number of experimental units low which are required for a confirmatory follow-up study. Results show that Bayesian statistical decision criteria can help in improving the reliability of preclinical animal research and should be considered more frequently in practice.

Mapping strategies towards improved external validity in preclinical translational research

INTRODUCTION

Translation is about successfully bringing findings from preclinical contexts into the clinic. This transfer is challenging as clinical trials frequently fail despite positive preclinical results. Limited robustness of preclinical research has been marked as one of the drivers of such failures. One suggested solution is to improve the external validity of in vitro and in vivo experiments via a suite of complementary strategies.

AREAS COVERED

In this review, the authors summarize the literature available on different strategies to improve external validity in in vivo, in vitro, or ex vivo experiments; systematic heterogenization; generalizability tests; and multi-batch and multicenter experiments. Articles that tested or discussed sources of variability in systematically heterogenized experiments were identified, and the most prevalent sources of variability are reviewed further. Special considerations in sample size planning, analysis options, and practical feasibility associated with each strategy are also reviewed.

EXPERT OPINION

The strategies reviewed differentially influence variation in experiments. Different research projects, with their unique goals, can leverage the strengths and limitations of each strategy. Applying a combination of these approaches in confirmatory stages of preclinical research putatively increases the chances of success in clinical studies.

Review of the Immune Checkpoint Inhibitors in the Context of Cancer Treatment

Checkpoint proteins are an integral part of the immune system and are used by the tumor cells to evade immune response, which helps them grow uncontrollably. By blocking these proteins, immune checkpoint inhibitors can restore the capability of the immune system to attack cancer cells and stop their growth. These findings are backed by adequate clinical trial data and presently, several FDA-approved immune checkpoint inhibitors exist in the market for treating various types of cancers, including melanoma, hepatocellular, endometrial, lung, kidney and others. Their mode of action is inhibition by targeting the checkpoint proteins CTLA-4, PD-1, PD-L1, etc. They can be used alone as well as in amalgamation with other cancer treatments, like surgery, radiation or chemotherapy. Since these drugs target only specific immune system proteins, their side effects are reduced in comparison with the traditional chemotherapy drugs, but may still cause a few affects like fatigue, skin rashes, and fever. In rare cases, these inhibitors are known to have caused more serious side effects, such as cardiotoxicity, and inflammation in the intestines or lungs. Herein, we provide an overview of these inhibitors and their role as biomarkers, immune-related adverse outcomes and clinical studies in the treatment of various cancers, as well as present some future perspectives.

Can acupuncture reverse oxidative stress and neuroinflammatory damage in animal models of vascular dementia?: A preclinical systematic review and meta-analysis

BACKGROUND

Vascular dementia is a cognitive dysfunction syndrome caused by cerebral vascular factors such as ischemic stroke and hemorrhagic stroke. The effect of acupuncture on vascular dementia models is ambiguous, and there is controversy about whether acupuncture has a placebo effect. Oxidative stress and inflammation are the most essential mechanisms in preclinical studies of vascular dementia. However, there is no meta-analysis on the mechanism of vascular dementia in animal models. It is necessary to explore the efficacy of acupuncture through Meta-analysis of preclinical studies.

METHODS

Three major databases, PubMed, Embase and Web of Science (including medline), were searched in English until December 2022.The quality of the including literature was assessed using SYRCLE's risk of bias tool. Review Manager 5.3 was used to statistically summarize the included studies and the statistical effect values were expressed by SMD. The outcomes included: behavioral tests (escape latency, number of crossings), pathological sections (Nissl and TUNEL staining), oxidative stress markers (ROS, MDA, SOD, GSH-PX) and neuroinflammatory factors (TNF-α, IL-1β, IL-6).

RESULTS

A total of 31 articles were included in this meta-analysis. The results showed that the escape latency, the contents of ROS, MDA, IL-1β, and IL-6 were decreased, and the contents of SOD and Nissl-positive neurons were increased in the acupuncture group as compared with the non-group (P < .05). Compared with the impaired group, the acupuncture group also had the above advantages (P < .05). In addition, the acupuncture group also increased the number of crossings and GSH-PX content, and decreased the expression of TUNEL-positive neurons and TNF-α (P < .05).

CONCLUSIONS

From behavioral tests to slices and pathological markers in animal models of vascular dementia, it can be proved that acupuncture is effective in targeting oxidative stress and neuroinflammatory damage, and acupuncture is not a placebo effect. Nevertheless, attention needs to be paid to the gap between animal experiments and clinical applications.

Preclinical trials in Alzheimer's disease: Sample size and effect size for behavioural and neuropathological outcomes in 5xFAD mice

5xFAD transgenic (TG) mice are used widely in AD preclinical trials; however, data on sample sizes are largely unaddressed. We therefore performed estimates of sample sizes and effect sizes for typical behavioural and neuropathological outcome measures in TG 5xFAD mice, based upon data from single-sex (female) groups. Group-size estimates to detect normalisation of TG body weight to WT littermate levels at 5.5m of age were N = 9-15 depending upon algorithm. However, by 1 year of age, group sizes were small (N = 1 -<6), likely reflecting the large difference between genotypes at this age. To detect normalisation of TG open-field hyperactivity to WT levels at 13-14m, group sizes were also small (N = 6-8). Cued learning in the Morris water maze (MWM) was normal in Young TG mice (5m of age). Mild deficits were noted during MWM spatial learning and memory. MWM reversal learning and memory revealed greater impairment, and groups of up to 22 TG mice were estimated to detect normalisation to WT performance. In contrast, Aged TG mice (tested between 13 and 14m) failed to complete the visual learning (non-spatial) phase of MWM learning, likely due to a failure to recognise the platform as an escape. Estimates of group size to detect normalisation of this severe impairment were small (N = 6-9, depending upon algorithm). Other cognitive tests including spontaneous and forced alternation and novel-object recognition either failed to reveal deficits in TG mice or deficits were negligible. For neuropathological outcomes, plaque load, astrocytosis and microgliosis in frontal cortex and hippocampus were quantified in TG mice aged 2m, 4m and 6m. Sample-size estimates were ≤9 to detect the equivalent of a reduction in plaque load to the level of 2m-old TG mice or the equivalent of normalisation of neuroinflammation outcomes. However, for a smaller effect size of 30%, larger groups of up to 21 mice were estimated. In light of published guidelines on preclinical trial design, these data may be used to provide provisional sample sizes and optimise preclinical trials in 5xFAD TG mice.

Statistical review of animal trials-A guideline

Any experiment involving living organisms requires justification of the need and moral defensibleness of the study. Statistical planning, design, and sample size calculation of the experiment are no less important review criteria than general medical and ethical points to consider. Errors made in the statistical planning and data evaluation phase can have severe consequences on both results and conclusions. They might proliferate and thus impact future trials-an unintended outcome of fundamental research with profound ethical consequences. Unified statistical standards are currently missing for animal review boards in Germany. In order to accompany, we developed a biometric form to be filled and handed in with the proposal at the concerned local authority on animal welfare. It addresses relevant points to consider for biostatistical planning of animal experiments and can help both the applicants and the reviewers in overseeing the entire experiment(s) planned. Furthermore, the form might also aid in meeting the current standards set by the 3+3R's principle of animal experimentation: Replacement, Reduction, Refinement as well as Robustness, Registration, and Reporting. The form has already been in use by the concerned local authority of animal welfare in Berlin, Germany. In addition, we provide reference to our user guide giving more detailed explanation and examples for each section of the biometric form. Unifying the set of biostatistical aspects will help both the applicants and the reviewers to equal standards and increase quality of preclinical research projects, also for translational, multicenter, or international studies.

Cell-3D matrix interactions: recent advances and opportunities

Tissues consist of cells and their surrounding extracellular matrix (ECM). Cell-ECM interactions play crucial roles in embryonic development, differentiation, tissue remodeling, and diseases including fibrosis and cancer. Recent research advances in characterizing cell-matrix interactions include detailed descriptions of hundreds of ECM and associated molecules, their complex intermolecular interactions in development and disease, identification of distinctive modes of cell migration in different 3D ECMs, and new insights into mechanisms of organ formation. Exploring the roles of the physical features of different ECM microenvironments and the bidirectional regulation of cell signaling and matrix organization emphasize the dynamic nature of these interactions, which can include feedback loops that exacerbate disease. Understanding mechanisms of cell-matrix interactions can potentially lead to targeted therapeutic interventions.

Questionable Research Practices, Low Statistical Power, and Other Obstacles to Replicability: Why Preclinical Neuroscience Research Would Benefit from Registered Reports

Replicability, the degree to which a previous scientific finding can be repeated in a distinct set of data, has been considered an integral component of institutionalized scientific practice since its inception several hundred years ago. In the past decade, large-scale replication studies have demonstrated that replicability is far from favorable, across multiple scientific fields. Here, I evaluate this literature and describe contributing factors including the prevalence of questionable research practices (QRPs), misunderstanding of p-values, and low statistical power. I subsequently discuss how these issues manifest specifically in preclinical neuroscience research. I conclude that these problems are multifaceted and difficult to solve, relying on the actions of early and late career researchers, funding sources, academic publishers, and others. I assert that any viable solution to the problem of substandard replicability must include changing academic incentives, with adoption of registered reports being the most immediately impactful and pragmatic strategy. For animal research in particular, comprehensive reporting guidelines that document potential sources of sensitivity for experimental outcomes is an essential addition.

Hamster models of COVID-19 pneumonia reviewed: How human can they be?

The dramatic global consequences of the coronavirus disease 2019 (COVID-19) pandemic soon fueled quests for a suitable model that would facilitate the development and testing of therapies and vaccines. In contrast to other rodents, hamsters are naturally susceptible to infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), and the Syrian hamster (Mesocricetus auratus) rapidly developed into a popular model. It recapitulates many characteristic features as seen in patients with a moderate, self-limiting course of the disease such as specific patterns of respiratory tract inflammation, vascular endothelialitis, and age dependence. Among 4 other hamster species examined, the Roborovski dwarf hamster (Phodopus roborovskii) more closely mimics the disease in highly susceptible patients with frequent lethal outcome, including devastating diffuse alveolar damage and coagulopathy. Thus, different hamster species are available to mimic different courses of the wide spectrum of COVID-19 manifestations in humans. On the other hand, fewer diagnostic tools and information on immune functions and molecular pathways are available than in mice, which limits mechanistic studies and inference to humans in several aspects. Still, under pandemic conditions with high pressure on progress in both basic and clinically oriented research, the Syrian hamster has turned into the leading non-transgenic model at an unprecedented pace, currently used in innumerable studies that all aim to combat the impact of the virus with its new variants of concern. As in other models, its strength rests upon a solid understanding of its similarities to and differences from the human disease, which we review here.

A multiparametric calcium signal screening platform using iPSC-derived cortical neural spheroids

Discovery of therapeutics for neurological diseases is hampered by the lack of predictive in vitro and in vivo models. Traditionally, in vitro assays rely on engineered cell lines grown two-dimensionally (2D) outside a physiological tissue context, which makes them very amenable for large scale drug screening but reduces their relevance to in vivo neurophysiology. In recent years, three-dimensional (3D) neural cell culture models derived from human induced pluripotent stem cells (iPSCs) have been developed as an in vitro assay platform to investigate brain development, neurological diseases, and for drug screening. iPSC-derived neural spheroids or organoids can be developed to include complex neuronal and glial cell populations and display spontaneous, synchronous activity, which is a hallmark of in vivo neural communication. In this report we present a proof-of-concept study evaluating 3D iPSC-derived cortical neural spheroids as a physiologically- and pharmacologically-relevant high-throughput screening (HTS) platform and investigate their potential for use for therapeutic development. To this end, a library of 687 neuroactive compounds were tested in a phenotypic screening paradigm which measured calcium activity as a functional biomarker for neural modulation through fluctuations in calcium fluorescence. Pharmacological responses of cortical neural spheroids were analyzed using a multi-parametric approach, whereby seven peak characteristics from the calcium activity in each well were quantified and incorporated into principal component analysis and Sammon mapping to measure compound response. Here, we describe the implementation of the 687-compound library screen and data analysis demonstrating that iPSC-derived cortical spheroids are a robust and information-rich assay platform for HTS.

Quantitative MRI Harmonization to Maximize Clinical Impact: The RIN-Neuroimaging Network

Neuroimaging studies often lack reproducibility, one of the cardinal features of the scientific method. Multisite collaboration initiatives increase sample size and limit methodological flexibility, therefore providing the foundation for increased statistical power and generalizable results. However, multisite collaborative initiatives are inherently limited by hardware, software, and pulse and sequence design heterogeneities of both clinical and preclinical MRI scanners and the lack of benchmark for acquisition protocols, data analysis, and data sharing. We present the overarching vision that yielded to the constitution of RIN-Neuroimaging Network, a national consortium dedicated to identifying disease and subject-specific in-vivo neuroimaging biomarkers of diverse neurological and neuropsychiatric conditions. This ambitious goal needs efforts toward increasing the diagnostic and prognostic power of advanced MRI data. To this aim, 23 Italian Scientific Institutes of Hospitalization and Care (IRCCS), with technological and clinical specialization in the neurological and neuroimaging field, have gathered together. Each IRCCS is equipped with high- or ultra-high field MRI scanners (i.e., ≥3T) for clinical or preclinical research or has established expertise in MRI data analysis and infrastructure. The actions of this Network were defined across several work packages (WP). A clinical work package (WP1) defined the guidelines for a minimum standard clinical qualitative MRI assessment for the main neurological diseases. Two neuroimaging technical work packages (WP2 and WP3, for clinical and preclinical scanners) established Standard Operative Procedures for quality controls on phantoms as well as advanced harmonized quantitative MRI protocols for studying the brain of healthy human participants and wild type mice. Under FAIR principles, a web-based e-infrastructure to store and share data across sites was also implemented (WP4). Finally, the RIN translated all these efforts into a large-scale multimodal data collection in patients and animal models with dementia (i.e., case study). The RIN-Neuroimaging Network can maximize the impact of public investments in research and clinical practice acquiring data across institutes and pathologies with high-quality and highly-consistent acquisition protocols, optimizing the analysis pipeline and data sharing procedures.

Do German university medical centres promote robust and transparent research? A cross-sectional study of institutional policies

BACKGROUND

In light of replication and translational failures, biomedical research practices have recently come under scrutiny. Experts have pointed out that the current incentive structures at research institutions do not sufficiently incentivise researchers to invest in robustness and transparency and instead incentivise them to optimize their fitness in the struggle for publications and grants. This cross-sectional study aimed to describe whether and how relevant policies of university medical centres in Germany support the robust and transparent conduct of research and how prevalent traditional metrics are.

METHODS

For 38 German university medical centres, we searched for institutional policies for academic degrees and academic appointments as well as websites for their core facilities and research in general between December 2020 and February 2021. We screened the documents for mentions of indicators of robust and transparent research (study registration; reporting of results; sharing of research data, code and protocols; open access; and measures to increase robustness) and for mentions of more traditional metrics of career progression (number of publications; number and value of awarded grants; impact factors; and authorship order).

RESULTS

While open access was mentioned in 16% of PhD regulations, other indicators of robust and transparent research were mentioned in less than 10% of institutional policies for academic degrees and academic appointments. These indicators were more frequently mentioned on the core facility and general research websites. Institutional policies for academic degrees and academic appointments had frequent mentions of traditional metrics.

CONCLUSIONS

References to robust and transparent research practices are, with a few exceptions, generally uncommon in institutional policies at German university medical centres, while traditional criteria for academic promotion and tenure still prevail.

Is preclinical research in cancer biology reproducible enough?

The Reproducibility Project: Cancer Biology (RPCB) was established to provide evidence about reproducibility in basic and preclinical cancer research, and to identify the factors that influence reproducibility more generally. In this commentary we address some of the scientific, ethical and policy implications of the project. We liken the basic and preclinical cancer research enterprise to a vast 'diagnostic machine' that is used to determine which clinical hypotheses should be advanced for further development, including clinical trials. The results of the RPCB suggest that this diagnostic machine currently recommends advancing many findings that are not reproducible. While concerning, we believe that more work needs to be done to evaluate the performance of the diagnostic machine. Specifically, we believe three questions remain unanswered: how often does the diagnostic machine correctly recommend against advancing real effects to clinical testing?; what are the relative costs to society of false positive and false negatives?; and how well do scientists and others interpret the outputs of the machine?

Is preclinical research in cancer biology reproducible enough?

The Reproducibility Project: Cancer Biology (RPCB) was established to provide evidence about reproducibility in basic and preclinical cancer research, and to identify the factors that influence reproducibility more generally. In this commentary we address some of the scientific, ethical and policy implications of the project. We liken the basic and preclinical cancer research enterprise to a vast 'diagnostic machine' that is used to determine which clinical hypotheses should be advanced for further development, including clinical trials. The results of the RPCB suggest that this diagnostic machine currently recommends advancing many findings that are not reproducible. While concerning, we believe that more work needs to be done to evaluate the performance of the diagnostic machine. Specifically, we believe three questions remain unanswered: how often does the diagnostic machine correctly recommend against advancing real effects to clinical testing?; what are the relative costs to society of false positive and false negatives?; and how well do scientists and others interpret the outputs of the machine?

Investigating the replicability of preclinical cancer biology

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.

Robust preclinical evidence in somatic cell genome editing: A key driver of responsible and efficient therapeutic innovations

Somatic cell genome editing (SCGE) is highly promising for therapeutic innovation. This study demonstrates that the majority of 46 preclinical SCGE studies discussed in reviews as particularly promising for clinical translation do not report on key elements for robust and confirmatory research practices: randomization, blinding, sample size calculation, data handling, pre-registration, multi-centric study design, and independent confirmation. We present the here-examined reporting standards and the new National Institutes of Health (NIH) funding criteria for SCGE research as a viable solution to protect this promising field from backlashes. We argue that the implementation of the novel methodological standards provides the opportunity for SCGE research to become a lighthouse example for trustworthy and useful translational research.

Epidemiology and reporting characteristics of preclinical systematic reviews

In an effort to better utilize published evidence obtained from animal experiments, systematic reviews of preclinical studies are increasingly more common-along with the methods and tools to appraise them (e.g., SYstematic Review Center for Laboratory animal Experimentation [SYRCLE's] risk of bias tool). We performed a cross-sectional study of a sample of recent preclinical systematic reviews (2015-2018) and examined a range of epidemiological characteristics and used a 46-item checklist to assess reporting details. We identified 442 reviews published across 43 countries in 23 different disease domains that used 26 animal species. Reporting of key details to ensure transparency and reproducibility was inconsistent across reviews and within article sections. Items were most completely reported in the title, introduction, and results sections of the reviews, while least reported in the methods and discussion sections. Less than half of reviews reported that a risk of bias assessment for internal and external validity was undertaken, and none reported methods for evaluating construct validity. Our results demonstrate that a considerable number of preclinical systematic reviews investigating diverse topics have been conducted; however, their quality of reporting is inconsistent. Our study provides the justification and evidence to inform the development of guidelines for conducting and reporting preclinical systematic reviews.