Investigating the replicability of preclinical cancer biology

Assessing the replicability of RCTs in RWE emulations

BACKGROUND

The standard regulatory approach to assess replication success is the two-trials rule, requiring both the original and the replication study to be significant with effect estimates in the same direction. The sceptical p-value was recently presented as an alternative method for the statistical assessment of the replicability of study results.

METHODS

We review the statistical properties of the sceptical p-value and compare those to the two-trials rule. We extend the methodology to non-inferiority trials and describe how to invert the sceptical p-value to obtain confidence intervals. We illustrate the performance of the different methods using real-world evidence emulations of randomized controlled trials (RCTs) conducted within the RCT DUPLICATE initiative.

RESULTS

The sceptical p-value depends not only on the two p-values, but also on sample size and effect size of the two studies. It can be calibrated to have the same Type-I error rate as the two-trials rule, but has larger power to detect an existing effect. In the application to the results from the RCT DUPLICATE initiative, the sceptical p-value leads to qualitatively similar results than the two-trials rule, but tends to show more evidence for treatment effects compared to the two-trials rule.

CONCLUSION

The sceptical p-value represents a valid statistical measure to assess the replicability of study results and is useful in the context of real-world evidence emulations.

International Brain Laboratory, Banga K, Benson J, Bhagat J, Biderman D, Birman D, Bonacchi N, Bruijns SA, Buchanan K, Campbell RAA, Carandini M, Chapuis GA, Churchland AK, Davatolhagh MF, Lee HD, Faulkner M, Gerçek B, Hu F, Huntenburg J, Hurwitz CL, Khanal A, Krasniak C, Lau P, Langfield C, Mackenzie N, Meijer GT, Miska NJ, Mohammadi Z, Noel JP, Paninski L, Pan-Vazquez A, Rossant C, Roth N, Schartner M, Socha KZ, Steinmetz NA, Svoboda K, Taheri M, Urai AE, Wang S, Wells M, West SJ, Whiteway MR, Winter O, Witten IB, Zhang Y. Reproducibility of in vivo electrophysiological measurements in mice

Understanding brain function relies on the collective work of many labs generating reproducible results. However, reproducibility has not been systematically assessed within the context of electrophysiological recordings during cognitive behaviors. To address this, we formed a multi-lab collaboration using a shared, open-source behavioral task and experimental apparatus. Experimenters in 10 laboratories repeatedly targeted Neuropixels probes to the same location (spanning secondary visual areas, hippocampus, and thalamus) in mice making decisions; this generated a total of 121 experimental replicates, a unique dataset for evaluating reproducibility of electrophysiology experiments. Despite standardizing both behavioral and electrophysiological procedures, some experimental outcomes were highly variable. A closer analysis uncovered that variability in electrode targeting hindered reproducibility, as did the limited statistical power of some routinely used electrophysiological analyses, such as single-neuron tests of modulation by individual task parameters. Reproducibility was enhanced by histological and electrophysiological quality-control criteria. Our observations suggest that data from systems neuroscience is vulnerable to a lack of reproducibility, but that across-lab standardization, including metrics we propose, can serve to mitigate this.

Alternative to the statistical mass confusion of testing for "no effect"

It should not be controversial to argue that the proximate goal of measuring something is to figure out how big or small or fast or slow it is. Estimates of effect size can be used to build models of how cells work and to test quantitative predictions. Unfortunately, in cell biology, quantification is nearly synonymous with null-hypothesis significance testing. The hypothesis being tested is universally assumed to be the hypothesis that there was no effect. Framing every experiment as an attempt to reject the no-effect hypothesis is convenient but doesn't teach us about cells. In this manuscript, I walk through some of the common critiques of significance testing and how these critiques relate to experimental cell biology. I argue that careful consideration of effect size should be returned to its central position in the planning and discussion of cell biological research. To facilitate this shift in focus, I recommend replacing p-values with confidence intervals as cell biology's default statistical analysis.

Replicability of bulk RNA-Seq differential expression and enrichment analysis results for small cohort sizes

The high-dimensional and heterogeneous nature of transcriptomics data from RNA sequencing (RNA-Seq) experiments poses a challenge to routine downstream analysis steps, such as differential expression analysis and enrichment analysis. Additionally, due to practical and financial constraints, RNA-Seq experiments are often limited to a small number of biological replicates. In light of recent studies on the low replicability of preclinical cancer research, it is essential to understand how the combination of population heterogeneity and underpowered cohort sizes affects the replicability of RNA-Seq research. Using 18'000 subsampled RNA-Seq experiments based on real gene expression data from 18 different data sets, we find that differential expression and enrichment analysis results from underpowered experiments are unlikely to replicate well. However, low replicability does not necessarily imply low precision of results, as data sets exhibit a wide range of possible outcomes. In fact, 10 out of 18 data sets achieve high median precision despite low recall and replicability for cohorts with more than five replicates. To assist researchers constrained by small cohort sizes in estimating the expected performance regime of their data sets, we provide a simple bootstrapping procedure that correlates strongly with the observed replicability and precision metrics. We conclude with practical recommendations to alleviate problems with underpowered RNA-Seq studies.

Cancer researchers' experiences with and perceptions of research data sharing: Results of a cross-sectional survey

BACKGROUND

Despite wide recognition of the benefits of sharing research data, public availability rates have not increased substantially in oncology or medicine more broadly over the last decade.

METHODS

We surveyed 285 cancer researchers to determine their prior experience with sharing data and views on known drivers and inhibitors.

RESULTS

We found that 45% of respondents had shared some data from their most recent empirical publication, with respondents who typically studied non-human research participants, or routinely worked with human genomic data, more likely to share than those who did not. A third of respondents added that they had previously shared data privately, with 74% indicating that doing so had also led to authorship opportunities or future collaborations for them. Journal and funder policies were reported to be the biggest general drivers toward sharing, whereas commercial interests, agreements with industrial sponsors and institutional policies were the biggest prohibitors. We show that researchers' decisions about whether to share data are also likely to be influenced by participants' desires.

CONCLUSIONS

Our survey suggests that increased promotion and support by research institutions, alongside greater championing of data sharing by journals and funders, may motivate more researchers in oncology to share their data.

Extraordinarily corrupt or statistically commonplace? Reproducibility crises may stem from a lack of understanding of outcome probabilities

Reports of crises of reproducibility have abounded in the scientific and popular press, and are often attributed to questionable research practices, lack of rigor in protocols, or fraud. On the other hand, it is a known fact that-just like observations in a single biological experiment-outcomes of biological replicates will vary; nevertheless, that variability is rarely assessed formally. Here I argue that some instances of failure to replicate experiments are in fact failures to properly describe the structure of variance. I formalize a hierarchy of distributions that represent the system-level and experiment-level effects, and correctly account for the between-and within-experiment variances, respectively. I also show that this formulation is straightforward to implement and generalize through Bayesian hierarchical models, although it doesn't preclude the use of Frequentist models. One of the main results of this approach is that a set of repetitions of an experiment, instead of being described by irreconcilable string of significant/nonsignificant results, are described and consolidated as a system-level distribution. As a corollary, stronger statements about a system can only be made by analyzing a number of replicates, so I argue that scientists should refrain from making them based on individual experiments.

Testing the reproducibility of ecological studies on insect behavior in a multi-laboratory setting identifies opportunities for improving experimental rigor

The reproducibility of studies involving insect species is an underexplored area in the broader discussion about poor reproducibility in science. Our study addresses this gap by conducting a systematic multi-laboratory investigation into the reproducibility of ecological studies on insect behavior. We implemented a 3 × 3 experimental design, incorporating three study sites, and three independent experiments on three insect species from different orders: the turnip sawfly (Athalia rosae, Hymenoptera), the meadow grasshopper (Pseudochorthippus parallelus, Orthoptera), and the red flour beetle (Tribolium castaneum, Coleoptera). Using random-effect meta-analysis, we compared the consistency and accuracy of treatment effects on insect behavioral traits across replicate experiments. We successfully reproduced the overall statistical treatment effect in 83% of the replicate experiments, but overall effect size replication was achieved in only 66% of the replicates. Thus, though demonstrating sufficient reproducibility in some measures, this study also provides the first experimental evidence for cases of poor reproducibility in insect experiments. Our findings further show that reasons causing poor reproducibility established in rodent research also hold for other study organisms and research questions. We believe that a rethinking of current best practices is required to face reproducibility issues in insect studies but also across disciplines. Specifically, we advocate for adopting open research practices and the implementation of methodological strategies that reduce bias and problems arising from over-standardization. With respect to the latter, the introduction of systematic variation through multi-laboratory or heterogenized designs may contribute to improved reproducibility in studies involving any living organisms.

Open science interventions to improve reproducibility and replicability of research: a scoping review

Various open science practices have been proposed to improve the reproducibility and replicability of scientific research, but not for all practices, there may be evidence they are indeed effective. Therefore, we conducted a scoping review of the literature on interventions to improve reproducibility. We systematically searched Medline, Embase, Web of Science, PsycINFO, Scopus and Eric, on 18 August 2023. Any study empirically evaluating the effectiveness of interventions aimed at improving the reproducibility or replicability of scientific methods and findings was included. We summarized the retrieved evidence narratively and in evidence gap maps. Of the 105 distinct studies we included, 15 directly measured the effect of an intervention on reproducibility or replicability, while the remainder addressed a proxy outcome that might be expected to increase reproducibility or replicability, such as data sharing, methods transparency or pre-registration. Thirty studies were non-comparative and 27 were comparative but cross-sectional observational designs, precluding any causal inference. Despite studies investigating a range of interventions and addressing various outcomes, our findings indicate that in general the evidence base for which various interventions to improve reproducibility of research remains remarkably limited in many respects.

Supporting evidence in phase 2 cancer trial protocols: a content analysis

BACKGROUND

Phase 2 trials are instrumental for designing definitive efficacy trials or attaining accelerated approval. However, high attrition of drug candidates in phase 2 trials raises questions about their supporting evidence.

METHODS

We developed a typology of supporting evidence for phase 2 cancer trials. We also devised a scheme for capturing elements that enable an assessment of the strength of such evidence. Using this framework, we content analyzed supporting evidence provided in protocols of 50 randomly sampled phase 2 cancer monotherapy trials between January 2014 and January 2019, available on ClinicalTrials.gov.

RESULTS

Of the 50 protocols in our sample, 52% were industry funded. Most invoked supporting evidence deriving from trials against different cancers (n = 28, 56%) or preclinical studies (n = 48, 96%) but not from clinical studies involving the target drug-indication pairing (n = 23, 46%). When presenting evidence from models, only 1 (2%) protocol explained its translational relevance. Instead, protocols implied translatability by describing molecular (86%) and pathophysiological (84%) processes shared by model and target systems. Protocols often provided information for assessing the magnitude, precision, and risk of bias for supporting trials (n = 43; 93%, 91%, 47%, respectively). However, such information was often unavailable for preclinical studies (n = 49; 53%, 22%, 59%, respectively).

CONCLUSIONS

Supporting evidence is key to justifying the commitment of scientific resources and patients to a clinical hypothesis. Protocols often omit elements that would enable critical assessment of supporting evidence for phase 2 monotherapy cancer trials. These gaps suggest the promise of more structured approaches for presenting supporting evidence.

Transparency of research practices in cardiovascular literature

Background

Several fields have described low reproducibility of scientific research and poor accessibility in research reporting practices. Although previous reports have investigated accessible reporting practices that lead to reproducible research in other fields, to date, no study has explored the extent of accessible and reproducible research practices in cardiovascular science literature.

Methods

To study accessibility and reproducibility in cardiovascular research reporting, we screened 639 randomly selected articles published in 2019 in three top cardiovascular science publications: Circulation, the European Heart Journal, and the Journal of the American College of Cardiology (JACC). Of those 639 articles, 393 were empirical research articles. We screened each paper for accessible and reproducible research practices using a set of accessibility criteria including protocol, materials, data, and analysis script availability, as well as accessibility of the publication itself. We also quantified the consistency of open research practices within and across cardiovascular study types and journal formats.

Results

We identified that fewer than 2% of cardiovascular research publications provide sufficient resources (materials, methods, data, and analysis scripts) to fully reproduce their studies. Of the 639 articles screened, 393 were empirical research studies for which reproducibility could be assessed using our protocol, as opposed to commentaries or reviews. After calculating an accessibility score as a measure of the extent to which an article makes its resources available, we also showed that the level of accessibility varies across study types with a score of 0.08 for case studies or case series and 0.39 for clinical trials (p = 5.500E-5) and across journals (0.19 through 0.34, p = 1.230E-2). We further showed that there are significant differences in which study types share which resources.

Conclusions

Although the degree to which reproducible reporting practices are present in publications varies significantly across journals and study types, current cardiovascular science reports frequently do not provide sufficient materials, protocols, data, or analysis information to reproduce a study. In the future, having higher standards of accessibility mandated by either journals or funding bodies will help increase the reproducibility of cardiovascular research.

Funding

Authors Gabriel Heckerman, Arely Campos-Melendez, and Chisomaga Ekwueme were supported by an NIH R25 grant from the National Heart Lung and Blood Institute (R25HL147666). Eileen Tzng was supported by an AHA Institutional Training Award fellowship (18UFEL33960207).

Transparency in the secondary use of health data: assessing the status quo of guidance and best practices

We evaluated what guidance exists in the literature to improve the transparency of studies that make secondary use of health data. To find peer-reviewed papers, we searched PubMed and Google Scholar. To find institutional documents, we used our personal expertise to draft a list of health organizations and searched their websites. We quantitatively and qualitatively coded different types of research transparency: registration, methods reporting, results reporting, data sharing and code sharing. We found 56 documents that provide recommendations to improve the transparency of studies making secondary use of health data, mainly in relation to study registration (n = 27) and/or methods reporting (n = 39). Only three documents made recommendations on data sharing or code sharing. Recommendations for study registration and methods reporting mainly came in the form of structured documents like registration templates and reporting guidelines. Aside from the recommendations aimed directly at researchers, we also found recommendations aimed at the wider research community, typically on how to improve research infrastructure. Limitations or challenges of improving transparency were rarely mentioned, highlighting the need for more nuance in providing transparency guidance for studies that make secondary use of health data.

How can we make sound replication decisions?

Replication and the reported crises impacting many fields of research have become a focal point for the sciences. This has led to reforms in publishing, methodological design and reporting, and increased numbers of experimental replications coordinated across many laboratories. While replication is rightly considered an indispensable tool of science, financial resources and researchers' time are quite limited. In this perspective, we examine different values and attitudes that scientists can consider when deciding whether to replicate a finding and how. We offer a conceptual framework for assessing the usefulness of various replication tools, such as preregistration.

Poor hypotheses and research waste in biology: learning from a theory crisis in psychology

While psychologists have extensively discussed the notion of a "theory crisis" arising from vague and incorrect hypotheses, there has been no debate about such a crisis in biology. However, biologists have long discussed communication failures between theoreticians and empiricists. We argue such failure is one aspect of a theory crisis because misapplied and misunderstood theories lead to poor hypotheses and research waste. We review its solutions and compare them with methodology-focused solutions proposed for replication crises. We conclude by discussing how promoting inclusion, diversity, equity, and accessibility (IDEA) in theoretical biology could contribute to ameliorating breakdowns in the theory-empirical cycle.

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

A systematic assessment of robustness in CNS safety pharmacology

BACKGROUND AND PURPOSE

Irwin tests are key preclinical study elements for characterising drug-induced neurological side effects. This multicentre study aimed to assess the robustness of Irwin tests across multinational sites during three stages of protocol harmonisation. The projects were part of the Enhanced Quality in Preclinical Data framework, aiming to increase success rates in transition from preclinical testing to clinical application.

EXPERIMENTAL APPROACH

Female and male NMRI mice were assigned to one of three groups (vehicle, MK-801 0.1 and 0.3 mg kg-1). Irwin scores were assessed at baseline and multiple times following intraperitoneal injection of MK-801 using local protocols (Stage 1), shared protocols with harmonised environmental design (Stage 2) and fully harmonised Irwin scoring protocols (Stage 3).

KEY RESULTS

The analysis based on the four functional domains (motor, autonomic, sedation and excitation) revealed substantial data variability in Stages 1 and 2. Although there was still marked overall heterogeneity between sites in Stage 3 after complete harmonisation of the Irwin scoring scheme, heterogeneity was only moderate within functional domains. When comparing treatment groups versus vehicle, we found large effect sizes in the motor domain and subtle to moderate effects in the excitation-related and autonomic domains.

CONCLUSION AND IMPLICATIONS

The pronounced interlaboratory variability in Irwin datasets for the CNS-active compound MK-801 needs to be carefully considered when making decisions during drug development. While environmental and general study design had a minor impact, the study suggests that harmonisation of parameters and their scoring can limit variability and increase robustness.

Improving Replication in Endometrial Omics: Understanding the Influence of the Menstrual Cycle

The dynamic nature of human endometrial tissue presents unique challenges in analysis. Despite extensive research into endometrial disorders such as endometriosis and infertility, recent systematic reviews have highlighted concerning issues with the reproducibility of omics studies attempting to identify biomarkers. This review examines factors contributing to poor reproducibility in endometrial omics research. Hormonal fluctuations in the menstrual cycle lead to widespread molecular changes in the endometrium, most notably in gene expression profiles. In this review, we examine the variability in omics data due to the menstrual cycle and highlight the importance of accurate menstrual cycle dating for effective statistical modelling. The current standards of endometrial dating lack precision and we make the case for using molecular-based modelling methods to estimate menstrual cycle time for endometrium tissue samples. Additionally, we discuss statistical considerations such as confounding and interaction effects, as well as the importance of recording the detailed and accurate clinical information of patients. By addressing these methodological challenges, we aim to establish more robust and reproducible research practises, increasing the reliability of endometrial omics research and biomarker discovery.

Risk of bias and problematic trials: characterising the research integrity of trials submitted to Anaesthesia

BACKGROUND

There is some evidence for systematic biases and failures of research integrity in the anaesthesia literature. However, the features of problematic trials and effect of editorial selection on these issues have not been well quantified.

METHODS

We analysed 209 randomised controlled trials submitted to Anaesthesia between 8 March 2019 and 31 March 2020. We evaluated the submitted manuscript, registry data and the results of investigations into the integrity of the trial undertaken at the time of submission. Trials were labelled 'concerning' if failures of research integrity were found, and 'problematic' if identified issues would have warranted retraction if they had been found after publication. We investigated how 'problematic' trials were detected, the distribution of p values and the risk of outcome reporting bias and p-hacking. We also investigated whether there were any factors that differed in problematic trials.

RESULTS

We found that false data was the most common reason for a trial to be labelled as 'concerning', which occurred in 51/62 (82%) cases. We also found that while 195/209 (93%) trials were preregistered, we found adequate registration for only 166/209 (79%) primary outcomes, 100/209 (48%) secondary outcomes and 11/209 (5%) analysis plans. We also found evidence for a step decrease in the frequency of p values > 0.05 compared with p values < 0.05. 'Problematic' trials were all single-centre and appeared to have fewer authors (incident risk ratio (95%CI) 0.8 (0.7-0.9)), but could not otherwise be distinguished reliably from other trials.

CONCLUSIONS

Identification of 'problematic' trials is frequently dependent on individual patient data, which is often unavailable after publication. Additionally, there is evidence of a risk of outcome reporting bias and p-hacking in submitted trials. Implementation of alternative research and editorial practices could reduce the risk of bias and make identification of problematic trials easier.

Anything goes for participant, patient and public involvement in youth mental health research

Background

Participant and Public Involvement in youth mental health research aims at making research more responsive to the needs of youth struggling with mental health issues, their parents, and mental health professionals and other stakeholders. Do characteristics of Patient and Public Involvement (PPI) in youth mental health research align with transparency and replication prerequisites as necessary conditions for translation? Relatedly, the question is addressed whether co-authorship should be assigned to youth involved in the study.

Methods

Here we address these questions re-visiting 50 PPI studies included in two recent systematic reviews of PPI on characteristics that are pertinent to questions about transparency, replicability, translatability, and co-authorship in PPI research.

Results

Almost two-third of the studies on youth mental health incorporating PPI translate their results to policy or practice, mostly as recommendations but sometimes also by dissemination of (online) interventions. At the same time the authors of a substantial majority of the studies (70%) also suggest the need for further work on their results, for example, in randomized controlled trials to validate the outcome of their exploratory inquiry. Only a quarter of the studies using PPI met the conditions for replicability, thus a majority of the PPI studies suggest premature translation of results. Authorship to involved participants was assigned in 24% of the studies.

Conclusions

"Anything goes" for PPI in an exploratory stage to generate fruitful hypotheses. Translation of the findings of PPI studies however require a firm evidence base of replicated results. Radical merging of research and action in participatory action research seems incompatible with replicable and therefore translatable inquiry. Assigning co-authorship to PPI representatives is often at odds with current guidelines for authorship. More evidence from randomized trials on the translational impact of PPI is needed before grant foundations should require PPI in grant proposals.

A large-scale in silico replication of ecological and evolutionary studies

Despite the growing concerns about the replicability of ecological and evolutionary studies, no results exist from a field-wide replication project. We conduct a large-scale in silico replication project, leveraging cutting-edge statistical methodologies. Replicability is 30%-40% for studies with marginal statistical significance in the absence of selective reporting, whereas the replicability of studies presenting 'strong' evidence against the null hypothesis H0 is >70%. The former requires a sevenfold larger sample size to reach the latter's replicability. We call for a change in planning, conducting and publishing research towards a transparent, credible and replicable ecology and evolution.

Transparency, bias, and reproducibility across science: a meta-research view

N/A.

Assessing and mitigating batch effects in large-scale omics studies

Batch effects in omics data are notoriously common technical variations unrelated to study objectives, and may result in misleading outcomes if uncorrected, or hinder biomedical discovery if over-corrected. Assessing and mitigating batch effects is crucial for ensuring the reliability and reproducibility of omics data and minimizing the impact of technical variations on biological interpretation. In this review, we highlight the profound negative impact of batch effects and the urgent need to address this challenging problem in large-scale omics studies. We summarize potential sources of batch effects, current progress in evaluating and correcting them, and consortium efforts aiming to tackle them.

Decatastrophizing research irreproducibility

The reported inability to replicate research findings from the published literature precipitated extensive efforts to identify and correct perceived deficiencies in the execution and reporting of biomedical research. Despite these efforts, quantification of the magnitude of irreproducible research or the effectiveness of associated remediation initiatives, across diverse biomedical disciplines, has made little progress over the last decade. The idea that science is self-correcting has been further challenged in recent years by the proliferation of unverified or fraudulent scientific content generated by predatory journals, paper mills, pre-print server postings, and the inappropriate use of artificial intelligence technologies. The degree to which the field of pharmacology has been negatively impacted by these evolving pressures is unknown. Regardless of these ambiguities, pharmacology societies and their associated journals have championed best practices to enhance the experimental rigor and reporting of pharmacological research. The value of transparent and independent validation of raw data generation and its analysis in basic and clinical research is exemplified by the discovery, development, and approval of Highly Effective Modulator Therapy (HEMT) for Cystic Fibrosis (CF) patients. This provides a didactic counterpoint to concerns regarding the current state of biomedical research. Key features of this important therapeutic advance include objective construction of basic and translational research hypotheses, associated experimental designs, and validation of experimental effect sizes with quantitative alignment to meaningful clinical endpoints with input from the FDA, which enhanced scientific rigor and transparency with real world deliverables for patients in need.

Replicable and robust cellular and biochemical blood marker signatures of depression and depressive symptoms

BACKGROUND

Identification of replicable and robust peripheral blood-based markers associated with depression remains elusive, given that studies frequently identify potential biomarkers that ultimately fail to replicate in other studies, impeding progress in psychiatric research. Peripheral biochemical and cellular markers (PBCs; e.g., albumin) may play an important role in depression.

METHODS

Using a test-replication design including participants from the NHANES community cohort (ntest=17,450, nreplication=17,449), we examined 42 PBCs to identify PBCs that were both replicably and robustly associated with either overall depression severity or individual symptoms of depression across both cohorts across a wide range of possible combinations of analytic decisions (n's = 17,000+).

RESULTS

We found that a small set of PBCs (e.g., bilirubin) were robustly and replicably associated with overall depression severity, with unique signatures of PBCs linked with individual symptoms of depression when stratified by gender. A varying degree of correlation was found between measures of replicability.

CONCLUSIONS

We identified replicable and robust cellular biochemical blood marker signatures associated with both overall depression severity and individual symptoms of depression. Our findings can be used to enhance other researchers' abilities to better understand factors associated with depression and potentially drive the development of effective treatments for depression.

Towards replicability and sustainability in cancer research

High-quality cancer research is crucial to both save lives and improve quality of life. Spurious findings, however, impedes these laudable goals by misleading research efforts and creating research waste that is inherently difficult to counteract. Irreproducible research is intrinsically wasteful, and unsustainable over the long term. In this perspective piece, we elucidate the extent of the current replication crisis and the underlying causes, identifying practices that lend themselves to unsustainable spurious findings, and the factors that underpin these practices. Finally we outline some remedies to the problem of irreproducible research, and how we might move towards more sustainable and trustworthy research in biomedical science.

Sharing Is Caring? International Society for Pharmacoepidemiology Review and Recommendations for Sharing Programming Code

PURPOSE

There is increasing recognition of the importance of transparency and reproducibility in scientific research. This study aimed to quantify the extent to which programming code is publicly shared in pharmacoepidemiology, and to develop a set of recommendations on this topic.

METHODS

We conducted a literature review identifying all studies published in Pharmacoepidemiology and Drug Safety (PDS) between 2017 and 2022. Data were extracted on the frequency and types of programming code shared, and other key open science practices (clinical codelist sharing, data sharing, study preregistration, and stated use of reporting guidelines and preprinting). We developed six recommendations for investigators who choose to share code and gathered feedback from members of the International Society for Pharmacoepidemiology (ISPE).

RESULTS

Programming code sharing by articles published in PDS ranged from 1.8% in 2017 to 9.5% in 2022. It was more prevalent among articles with a methodological focus, simulation studies, and papers which also shared record-level data.

CONCLUSION

Programming code sharing is rare but increasing in pharmacoepidemiology studies published in PDS. We recommend improved reporting of whether code is shared and how available code can be accessed. When sharing programming code, we recommend the use of permanent digital identifiers, appropriate licenses, and, where possible, adherence to good software practices around the provision of metadata and documentation, computational reproducibility, and data privacy.

OBSERVE: guidelines for the refinement of rodent cancer models

Existing guidelines on the preparation (Planning Research and Experimental Procedures on Animals: Recommendations for Excellence (PREPARE)) and reporting (Animal Research: Reporting of In Vivo Experiments (ARRIVE)) of animal experiments do not provide a clear and standardized approach for refinement during in vivo cancer studies, resulting in the publication of generic methodological sections that poorly reflect the attempts made at accurately monitoring different pathologies. Compliance with the 3Rs guidelines has mainly focused on reduction and replacement; however, refinement has been harder to implement. The Oncology Best-practices: Signs, Endpoints and Refinements for in Vivo Experiments (OBSERVE) guidelines are the result of a European initiative supported by EurOPDX and INFRAFRONTIER, and aim to facilitate the refinement of studies using in vivo cancer models by offering robust and practical recommendations on approaches to research scientists and animal care staff. We listed cancer-specific clinical signs as a reference point and from there developed sets of guidelines for a wide variety of rodent models, including genetically engineered models and patient derived xenografts. In this Consensus Statement, we systematically and comprehensively address refinement and monitoring approaches during the design and execution of murine cancer studies. We elaborate on the appropriate preparation of tumor-initiating biologicals and the refinement of tumor-implantation methods. We describe the clinical signs to monitor associated with tumor growth, the appropriate follow-up of animals tailored to varying clinical signs and humane endpoints, and an overview of severity assessment in relation to clinical signs, implantation method and tumor characteristics. The guidelines provide oncology researchers clear and robust guidance for the refinement of in vivo cancer models.

CheckMyMetal (CMM): validating metal-binding sites in X-ray and cryo-EM data

Identifying and characterizing metal-binding sites (MBS) within macromolecular structures is imperative for elucidating their biological functions. CheckMyMetal (CMM) is a web based tool that facilitates the interactive validation of MBS in structures determined through X-ray crystallography and cryo-electron microscopy (cryo-EM). Recent updates to CMM have significantly enhanced its capability to efficiently handle large datasets generated from cryo-EM structural analyses. In this study, we address various challenges inherent in validating MBS within both X-ray and cryo-EM structures. Specifically, we examine the difficulties associated with accurately identifying metals and modeling their coordination environments by considering the ongoing reproducibility challenges in structural biology and the critical importance of well annotated, high-quality experimental data. CMM employs a sophisticated framework of rules rooted in the valence bond theory for MBS validation. We explore how CMM validation parameters correlate with the resolution of experimentally derived structures of macromolecules and their complexes. Additionally, we showcase the practical utility of CMM by analyzing a representative cryo-EM structure. Through a comprehensive examination of experimental data, we demonstrate the capability of CMM to advance MBS characterization and identify potential instances of metal misassignment.

ChatGPT as Research Scientist: Probing GPT's capabilities as a Research Librarian, Research Ethicist, Data Generator, and Data Predictor

How good a research scientist is ChatGPT? We systematically probed the capabilities of GPT-3.5 and GPT-4 across four central components of the scientific process: as a Research Librarian, Research Ethicist, Data Generator, and Novel Data Predictor, using psychological science as a testing field. In Study 1 (Research Librarian), unlike human researchers, GPT-3.5 and GPT-4 hallucinated, authoritatively generating fictional references 36.0% and 5.4% of the time, respectively, although GPT-4 exhibited an evolving capacity to acknowledge its fictions. In Study 2 (Research Ethicist), GPT-4 (though not GPT-3.5) proved capable of detecting violations like p-hacking in fictional research protocols, correcting 88.6% of blatantly presented issues, and 72.6% of subtly presented issues. In Study 3 (Data Generator), both models consistently replicated patterns of cultural bias previously discovered in large language corpora, indicating that ChatGPT can simulate known results, an antecedent to usefulness for both data generation and skills like hypothesis generation. Contrastingly, in Study 4 (Novel Data Predictor), neither model was successful at predicting new results absent in their training data, and neither appeared to leverage substantially new information when predicting more vs. less novel outcomes. Together, these results suggest that GPT is a flawed but rapidly improving librarian, a decent research ethicist already, capable of data generation in simple domains with known characteristics but poor at predicting novel patterns of empirical data to aid future experimentation.

Estimating the replicability of highly cited clinical research (2004-2018)

INTRODUCTION

Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.

METHODS

We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.

RESULTS

A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.

CONCLUSION

Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.

The assessment of replicability using the sum of p-values

Statistical significance of both the original and the replication study is a commonly used criterion to assess replication attempts, also known as the two-trials rule in drug development. However, replication studies are sometimes conducted although the original study is non-significant, in which case Type-I error rate control across both studies is no longer guaranteed. We propose an alternative method to assess replicability using the sum of p -values from the two studies. The approach provides a combined p -value and can be calibrated to control the overall Type-I error rate at the same level as the two-trials rule but allows for replication success even if the original study is non-significant. The unweighted version requires a less restrictive level of significance at replication if the original study is already convincing which facilitates sample size reductions of up to 10%. Downweighting the original study accounts for possible bias and requires a more stringent significance level and larger sample sizes at replication. Data from four large-scale replication projects are used to illustrate and compare the proposed method with the two-trials rule, meta-analysis and Fisher's combination method.

Memory effects of prior subculture may impact the quality of multiomic perturbation profiles

Mass spectrometry-based omics technologies are increasingly used in perturbation studies to map drug effects to biological pathways by identifying significant molecular events. Significance is influenced by fold change and variation of each molecular parameter, but also by multiple testing corrections. While the fold change is largely determined by the biological system, the variation is determined by experimental workflows. Here, it is shown that memory effects of prior subculture can influence the variation of perturbation profiles using the two colon carcinoma cell lines SW480 and HCT116. These memory effects are largely driven by differences in growth states that persist into the perturbation experiment. In SW480 cells, memory effects combined with moderate treatment effects amplify the variation in multiple omics levels, including eicosadomics, proteomics, and phosphoproteomics. With stronger treatment effects, the memory effect was less pronounced, as demonstrated in HCT116 cells. Subculture homogeneity was controlled by real-time monitoring of cell growth. Controlled homogeneous subculture resulted in a perturbation network of 321 causal conjectures based on combined proteomic and phosphoproteomic data, compared to only 58 causal conjectures without controlling subculture homogeneity in SW480 cells. Some cellular responses and regulatory events were identified that extend the mode of action of arsenic trioxide (ATO) only when accounting for these memory effects. Controlled prior subculture led to the finding of a synergistic combination treatment of ATO with the thioredoxin reductase 1 inhibitor auranofin, which may prove useful in the management of NRF2-mediated resistance mechanisms.

Transparency and Reproducibility in the Adolescent Brain Cognitive Development (ABCD) Study

Background

Transparency can build trust in the scientific process, but scientific findings can be undermined by poor and obscure data use and reporting practices. The purpose of this work is to report how data from the Adolescent Brain Cognitive Development (ABCD) Study has been used to date, and to provide practical recommendations on how to improve the transparency and reproducibility of findings.

Methods

Articles published from 2017 to 2023 that used ABCD Study data were reviewed using more than 30 data extraction items to gather information on data use practices. Total frequencies were reported for each extraction item, along with computation of a Level of Completeness (LOC) score that represented overall endorsement of extraction items. Univariate linear regression models were used to examine the correlation between LOC scores and individual extraction items. Post hoc analysis included examination of whether LOC scores were correlated with the logged 2-year journal impact factor.

Results

There were 549 full-length articles included in the main analysis. Analytic scripts were shared in 30% of full-length articles. The number of participants excluded due to missing data was reported in 60% of articles, and information on missing data for individual variables (e.g., household income) was provided in 38% of articles. A table describing the analytic sample was included in 83% of articles. A race and/or ethnicity variable was included in 78% of reviewed articles, while its inclusion was justified in only 41% of these articles. LOC scores were highly correlated with extraction items related to examination of missing data. A bottom 10% of LOC score was significantly correlated with a lower logged journal impact factor when compared to the top 10% of LOC scores (β=-0.77, 95% -1.02, -0.51; p-value < 0.0001).

Conclusion

These findings highlight opportunities for improvement in future papers using ABCD Study data to readily adapt analytic practices for better transparency and reproducibility efforts. A list of recommendations is provided to facilitate adherence in future research.

Replication of null results: Absence of evidence or evidence of absence?

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a 'replication success.' Here, we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and 'replication success' can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with 'null results' are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.

Publishing important work that lacks validity or reproducibility - pushing frontiers or corrupting science?

Scientific research requires objectivity, impartiality and stringency. However, scholarly literature is littered with preliminary and explorative findings that lack reproducibility or validity. Some low-quality papers with perceived high impact have become publicly notable. The collective effort of fellow researchers who follow these false leads down blind alleys and impasses is a waste of time and resources, and this is particularly damaging for early career researchers. Furthermore, the lay public might also be affected by socioeconomic repercussions associated with the findings. It is arguable that the nature of scientific research is such that its frontiers are moved and shaped by cycles of published claims inducing in turn rounds of validation by others. Using recent example cases of room-temperature superconducting materials research, I argue instead that publication of perceptibly important or spectacular claims that lack reproducibility or validity is epistemically and socially irresponsible. This is even more so if authors refuse to share research materials and raw data for verification by others. Such acts do not advance, but would instead corrupt science, and should be prohibited by consensual governing rules on material and data sharing within the research community, with malpractices appropriately sanctioned.

Twelve years after the ARRIVE guidelines: Animal research has not yet arrived at high standards

The reproducibility crisis across animal studies jeopardizes the credibility of the main findings derived from animal research, even though these findings are critical for informing human studies. To clarify and improve transparency among animal studies, the ARRIVE reporting guidelines were first announced in 2010 and upgraded to version 2.0 in 2020. However, compliance with and awareness of those reporting guidelines has remained suboptimal. Journal editors should encourage the authors to adhere to those guidelines. Authors, editors, referees, and reviewers should be aware of the ARRIVE guideline 2.0 when assessing and evaluating the methodology and findings of animal studies. However, we should also question whether reporting guidelines alone can change a research culture and improve the reproducibility of animal investigations. Reported research may not reflect actual research. Large segments of animal research efforts are wasted because of poor design choices and because of non-publication rather than suboptimal reporting. Better training of the scientific workforce, interventions at improving animal research at the design stage, registration practices, and alignment of the reward system with the publication of rigorous animal research may achieve more than reporting guidelines alone.

Bounded research ethicality: researchers rate themselves and their field as better than others at following good research practice

Bounded ethicality refers to people's limited capacity to consistently behave in line with their ethical standards. Here, we present results from a pre-registered, large-scale (N = 11,050) survey of researchers in Sweden, suggesting that researchers too are boundedly ethical. Specifically, researchers on average rated themselves as better than other researchers in their field at following good research practice, and rated researchers in their own field as better than researchers in other fields at following good research practice. These effects were stable across all academic fields, but strongest among researchers in the medical sciences. Taken together, our findings illustrate inflated self-righteous beliefs among researchers and research disciplines when it comes to research ethics, which may contribute to academic polarization and moral blindspots regarding one's own and one's colleagues' use of questionable research practices.

High replicability of newly discovered social-behavioural findings is achievable

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

Promoting research quality

Plenty of awards recognize scientific contributions, but a unique and important one honors those whose efforts significantly enhance the quality and robustness of research. We discuss why this is important to promote trust in science.

Artificial Intelligence for Drug Discovery: Are We There Yet?

Drug discovery is adapting to novel technologies such as data science, informatics, and artificial intelligence (AI) to accelerate effective treatment development while reducing costs and animal experiments. AI is transforming drug discovery, as indicated by increasing interest from investors, industrial and academic scientists, and legislators. Successful drug discovery requires optimizing properties related to pharmacodynamics, pharmacokinetics, and clinical outcomes. This review discusses the use of AI in the three pillars of drug discovery: diseases, targets, and therapeutic modalities, with a focus on small-molecule drugs. AI technologies, such as generative chemistry, machine learning, and multiproperty optimization, have enabled several compounds to enter clinical trials. The scientific community must carefully vet known information to address the reproducibility crisis. The full potential of AI in drug discovery can only be realized with sufficient ground truth and appropriate human intervention at later pipeline stages.

Robustness and reproducibility for AI learning in biomedical sciences: RENOIR

Artificial intelligence (AI) techniques are increasingly applied across various domains, favoured by the growing acquisition and public availability of large, complex datasets. Despite this trend, AI publications often suffer from lack of reproducibility and poor generalisation of findings, undermining scientific value and contributing to global research waste. To address these issues and focusing on the learning aspect of the AI field, we present RENOIR (REpeated random sampliNg fOr machIne leaRning), a modular open-source platform for robust and reproducible machine learning (ML) analysis. RENOIR adopts standardised pipelines for model training and testing, introducing elements of novelty, such as the dependence of the performance of the algorithm on the sample size. Additionally, RENOIR offers automated generation of transparent and usable reports, aiming to enhance the quality and reproducibility of AI studies. To demonstrate the versatility of our tool, we applied it to benchmark datasets from health, computer science, and STEM (Science, Technology, Engineering, and Mathematics) domains. Furthermore, we showcase RENOIR's successful application in recently published studies, where it identified classifiers for SET2D and TP53 mutation status in cancer. Finally, we present a use case where RENOIR was employed to address a significant pharmacological challenge-predicting drug efficacy. RENOIR is freely available at https://github.com/alebarberis/renoir .

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

Behavioral Analysis of NMDAR Function in Rodents: Tests of Long-Term Spatial Memory

NMDAR-dependent forms of synaptic plasticity in brain regions like the hippocampus are widely believed to provide the neural substrate for long-term associative memory formation. However, the experimental data are equivocal at best and may suggest a more nuanced role for NMDARs and synaptic plasticity in memory. Much of the experimental data available comes from studies in genetically modified mice in which NMDAR subunits have been deleted or mutated in order to disrupt NMDAR function. Behavioral assessment of long-term memory in these mice has involved tests like the Morris watermaze and the radial arm maze. Here we describe these behavioral tests and some of the different testing protocols that can be used to assess memory performance. We discuss the importance of distinguishing selective effects on learning and memory processes from nonspecific effects on sensorimotor or motivational aspects of performance.

Current Practices in LC-MS Untargeted Metabolomics: A Scoping Review on the Use of Pooled Quality Control Samples

Untargeted metabolomics is an analytical approach with numerous applications serving as an effective metabolic phenotyping platform to characterize small molecules within a biological system. Data quality can be challenging to evaluate and demonstrate in metabolomics experiments. This has driven the use of pooled quality control (QC) samples for monitoring and, if necessary, correcting for analytical variance introduced during sample preparation and data acquisition stages. Described herein is a scoping literature review detailing the use of pooled QC samples in published untargeted liquid chromatography-mass spectrometry (LC-MS) based metabolomics studies. A literature query was performed, the list of papers was filtered, and suitable articles were randomly sampled. In total, 109 papers were each reviewed by at least five reviewers, answering predefined questions surrounding the use of pooled quality control samples. The results of the review indicate that use of pooled QC samples has been relatively widely adopted by the metabolomics community and that it is used at a similar frequency across biological taxa and sample types in both small- and large-scale studies. However, while many studies generated and analyzed pooled QC samples, relatively few reported the use of pooled QC samples to improve data quality. This demonstrates a clear opportunity for the field to more frequently utilize pooled QC samples for quality reporting, feature filtering, analytical drift correction, and metabolite annotation. Additionally, our survey approach enabled us to assess the ambiguity in the reporting of the methods used to describe the generation and use of pooled QC samples. This analysis indicates that many details of the QC framework are missing or unclear, limiting the reader's ability to determine which QC steps have been taken. Collectively, these results capture the current state of pooled QC sample usage and highlight existing strengths and deficiencies as they are applied in untargeted LC-MS metabolomics.

Preferred Reporting Items for Resistance Exercise Studies (PRIRES): A Checklist Developed Using an Umbrella Review of Systematic Reviews

BACKGROUND

The issues of replication and scientific transparency have been raised in exercise and sports science research. A potential means to address the replication crisis and enhance research reliability is to improve reporting quality and transparency. This study aims to formulate a reporting checklist as a supplement to the existing reporting guidelines, specifically for resistance exercise studies.

METHODS

PubMed (which covers Medline) and Scopus (which covers Medline, EMBASE, Ei Compendex, World Textile Index, Fluidex, Geobase, Biobase, and most journals in Web of Science) were searched for systematic reviews that comprised the primary studies directly comparing different resistance training methods. Basic data on the selected reviews, including on authors, publication years, and objectives, were summarized. The reporting items for the checklist were identified based on the objective of the reviews. Additional items from an existing checklist, namely the Consensus on Exercise Reporting Template, a National Strength and Conditioning Association handbook, and an article from the EQUATOR library were incorporated into the final reporting checklist.

RESULTS

Our database search retrieved 3595 relevant records. After automatic duplicate removal, the titles and abstracts of the remaining 2254 records were screened. The full texts of 137 records were then reviewed, and 88 systematic reviews that met the criteria were included in the umbrella review.

CONCLUSION

Developed primarily by an umbrella review method, this checklist covers the research questions which have been systematically studied and is expected to improve the reporting completeness of future resistance exercise studies. The PRIRES checklist comprises 26 reporting items (39 subitems) that cover four major topics in resistance exercise intervention: 1) exercise selection, performance, and training parameters, 2) training program and progression, 3) exercise setting, and 4) planned vs actual training. The PRIRES checklist was designed specifically for reporting resistance exercise intervention. It is expected to be used with other reporting guidelines such as Consolidated Standards of Reporting Trials and Standard Protocol Items: Recommendations for Interventional Trials. This article presents only the development process and resulting items of the checklist. An accompanying article detailing the rationale for, the importance of, and examples of each item is being prepared.

REGISTRATION

This study is registered with the EQUATOR Network under the title "Preferred Reporting Items for Resistance Exercise Studies (PRIRES)." PROSPERO registration number: CRD42021235259.

Experience of irreproducibility as a risk factor for poor mental health in biomedical science doctoral students: A survey and interview-based study

High rates of irreproducibility and of poor mental health in graduate students have been reported in the biomedical sciences in the past ten years, but to date, little research has investigated whether these two trends interact. In this study, we ask whether the experience of failing to replicate an expected finding impacts graduate students' mental health. Using an online survey paired with semi-structured qualitative interviews, we examined how often biomedical science doctoral students at a large American public university experienced events that could be interpreted as failures to replicate and how they responded to these experiences. We found that almost all participants had experience with irreproducibility: 84% had failed to replicate their own results, 70% had failed to replicate a colleague's finding, and 58% had failed to replicate a result from the published literature. Participants reported feelings of self-doubt, frustration, and depression while experiencing irreproducibility, and in 24% of cases, these emotional responses were strong enough to interfere with participants' eating, sleeping, or ability to work. A majority (82%) of participants initially believed that the anomalous results could be attributed to their own error. However, after further experimentation, most participants concluded that the original result was wrong (38%), that there was a key difference between the original experiment and their own (17%), or that there was a problem with the protocol (17%). These results suggest that biomedical science graduate students may be biased towards initially interpreting failures to replicate as indicative of a lack of skill, which may trigger or perpetuate feelings of anxiety, depression, or impostorism.