Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology

The EstroGene Database Reveals Diverse Temporal, Context-Dependent, and Bidirectional Estrogen Receptor Regulomes in Breast Cancer

As one of the most successful cancer therapeutic targets, estrogen receptor-α (ER/ESR1) has been extensively studied over the past few decades. Sequencing technological advances have enabled genome-wide analysis of ER action. However, comparison of individual studies is limited by different experimental designs, and few meta-analyses are available. Here, we established the EstroGene database through unified processing of data from 246 experiments including 136 transcriptomic, cistromic, and epigenetic datasets focusing on estradiol (E2)-triggered ER activation across 19 breast cancer cell lines. A user-friendly browser (https://estrogene.org/) was generated for multiomic data visualization involving gene inquiry under user-defined experimental conditions and statistical thresholds. Notably, annotation of metadata associated with public datasets revealed a considerable lack of experimental details. Comparison of independent RNA-seq or ER ChIP-seq data with the same design showed large variability and only strong effects could be consistently detected. Temporal estrogen response metasignatures were defined, and the association of E2 response rate with temporal transcriptional factors, chromatin accessibility, and heterogeneity of ER expression was evaluated. Unexpectedly, harmonizing 146 E2-induced transcriptomic datasets uncovered a subset of genes harboring bidirectional E2 regulation, which was linked to unique transcriptional factors and highly associated with immune surveillance in the clinical setting. Furthermore, the context dependent E2 response programs were characterized in MCF7 and T47D cell lines, the two most frequently used models in the EstroGene database. Collectively, the EstroGene database provides an informative and practical resource to the cancer research community to uniformly evaluate key reproducible features of ER regulomes and unravels modes of ER signaling.

SIGNIFICANCE

A resource database integrating 246 publicly available ER profiling datasets facilitates meta-analyses and identifies estrogen response temporal signatures, a bidirectional program, and model-specific biases.

EstroGene database reveals diverse temporal, context-dependent and directional estrogen receptor regulomes in breast cancer

As one of the most successful cancer therapeutic targets, estrogen receptor-α (ER/ESR1) has been extensively studied in decade-long. Sequencing technological advances have enabled genome-wide analysis of ER action. However, reproducibility is limited by different experimental design. Here, we established the EstroGene database through centralizing 246 experiments from 136 transcriptomic, cistromic and epigenetic datasets focusing on estradiol-treated ER activation across 19 breast cancer cell lines. We generated a user-friendly browser ( https://estrogene.org/ ) for data visualization and gene inquiry under user-defined experimental conditions and statistical thresholds. Notably, documentation-based meta-analysis revealed a considerable lack of experimental details. Comparison of independent RNA-seq or ER ChIP-seq data with the same design showed large variability and only strong effects could be consistently detected. We defined temporal estrogen response metasignatures and showed the association with specific transcriptional factors, chromatin accessibility and ER heterogeneity. Unexpectedly, harmonizing 146 transcriptomic analyses uncovered a subset of E2-bidirectionally regulated genes, which linked to immune surveillance in the clinical setting. Furthermore, we defined context dependent E2 response programs in MCF7 and T47D cell lines, the two most frequently used models in the field. Collectively, the EstroGene database provides an informative resource to the cancer research community and reveals a diverse mode of ER signaling.

Large inherent variability in data derived from highly standardised cell culture experiments

Cancer drug development is hindered by high clinical attrition rates, which are blamed on weak predictive power by preclinical models and limited replicability of preclinical findings. However, the technically feasible level of replicability remains unknown. To fill this gap, we conducted an analysis of data from the NCI60 cancer cell line screen (2.8 million compound/cell line experiments), which is to our knowledge the largest depository of experiments that have been repeatedly performed over decades. The findings revealed profound intra-laboratory data variability, although all experiments were executed following highly standardised protocols that avoid all known confounders of data quality. All compound/ cell line combinations with > 100 independent biological replicates displayed maximum GI50 (50% growth inhibition) fold changes (highest/ lowest GI50) > 5% and 70.5% displayed maximum fold changes > 1000. The highest maximum fold change was 3.16 × 10¹⁰ (lowest GI50: 7.93 ×10^-10 µM, highest GI50: 25.0 µM). FDA-approved drugs and experimental agents displayed similar variation. Variability remained high after outlier removal, when only considering experiments that tested drugs at the same concentration range, and when only considering NCI60-provided quality-controlled data. In conclusion, high variability is an intrinsic feature of anti-cancer drug testing, even among standardised experiments in a world-leading research environment. Awareness of this inherent variability will support realistic data interpretation and inspire research to improve data robustness. Further research will have to show whether the inclusion of a wider variety of model systems, such as animal and/ or patient-derived models, may improve data robustness.

From the Catastrophic Objective Irreproducibility of Cancer Research and Unavoidable Failures of Molecular Targeted Therapies to the Sparkling Hope of Supramolecular Targeted Strategies

The unprecedented non-reproducibility of the results published in the field of cancer research has recently come under the spotlight. In this short review, we try to highlight some general principles in the organization and evolution of cancerous tumors, which objectively lead to their enormous variability and, consequently, the irreproducibility of the results of their investigation. This heterogeneity is also extremely unfavorable for the effective use of molecularly targeted medicine. Against the seemingly comprehensive background of this heterogeneity, we single out two supramolecular characteristics common to all tumors: the clustered nature of tumor interactions with their microenvironment and the formation of biomolecular condensates with tumor-specific distinctive features. We suggest that these features can form the basis of strategies for tumor-specific supramolecular targeted therapies.

Selenium Modulates Cancer Cell Response to Pharmacologic Ascorbate

High-dose ascorbate (vitamin C) has shown promising anticancer activity. Two redox mechanisms have been proposed: hydrogen peroxide generation by ascorbate itself or glutathione depletion by dehydroascorbate (formed by ascorbate oxidation). Here we show that the metabolic effects and cytotoxicity of high-dose ascorbate in vitro result from hydrogen peroxide independently of dehydroascorbate. These effects were suppressed by selenium through antioxidant selenoenzymes including glutathione peroxidase 1 (GPX1) but not the classic ferroptosis-inhibiting selenoenzyme GPX4. Selenium-mediated protection from ascorbate was powered by NADPH from the pentose phosphate pathway. In vivo, dietary selenium deficiency resulted in significant enhancement of ascorbate activity against glioblastoma xenografts. These data establish selenoproteins as key mediators of cancer redox homeostasis. Cancer sensitivity to free radical-inducing therapies, including ascorbate, may depend on selenium, providing a dietary approach for improving their anticancer efficacy.

SIGNIFICANCE

Selenium restriction augments ascorbate efficacy and extends lifespan in a mouse xenograft model of glioblastoma, suggesting that targeting selenium-mediated antioxidant defenses merits clinical evaluation in combination with ascorbate and other pro-oxidant therapies.

Data Incompleteness May form a Hard-to-Overcome Barrier to Decoding Life's Mechanism

In this brief review, we attempt to demonstrate that the incompleteness of data, as well as the intrinsic heterogeneity of biological systems, may form very strong and possibly insurmountable barriers for researchers trying to decipher the mechanisms of the functioning of live systems. We illustrate this challenge using the two most studied organisms: E. coli, with 34.6% genes lacking experimental evidence of function, and C. elegans, with identified proteins for approximately 50% of its genes. Another striking example is an artificial unicellular entity named JCVI-syn3.0, with a minimal set of genes. A total of 31.5% of the genes of JCVI-syn3.0 cannot be ascribed a specific biological function. The human interactome mapping project identified only 5-10% of all protein interactions in humans. In addition, most of the available data are static snapshots, and it is barely possible to generate realistic models of the dynamic processes within cells. Moreover, the existing interactomes reflect the de facto interaction but not its functional result, which is an unpredictable emerging property. Perhaps the completeness of molecular data on any living organism is beyond our reach and represents an unsolvable problem in biology.

Testing the reproducibility and robustness of the cancer biology literature by robot

Scientific results should not just be ‘repeatable’ (replicable in the same laboratory under identical conditions), but also ‘reproducible’ (replicable in other laboratories under similar conditions). Results should also, if possible, be ‘robust’ (replicable under a wide range of conditions). The reproducibility and robustness of only a small fraction of published biomedical results has been tested; furthermore, when reproducibility is tested, it is often not found. This situation is termed ‘the reproducibility crisis', and it is one the most important issues facing biomedicine. This crisis would be solved if it were possible to automate reproducibility testing. Here, we describe the semi-automated testing for reproducibility and robustness of simple statements (propositions) about cancer cell biology automatically extracted from the literature. From 12 260 papers, we automatically extracted statements predicted to describe experimental results regarding a change of gene expression in response to drug treatment in breast cancer, from these we selected 74 statements of high biomedical interest. To test the reproducibility of these statements, two different teams used the laboratory automation system Eve and two breast cancer cell lines (MCF7 and MDA-MB-231). Statistically significant evidence for repeatability was found for 43 statements, and significant evidence for reproducibility/robustness in 22 statements. In two cases, the automation made serendipitous discoveries. The reproduced/robust knowledge provides significant insight into cancer. We conclude that semi-automated reproducibility testing is currently achievable, that it could be scaled up to generate a substantive source of reliable knowledge and that automation has the potential to mitigate the reproducibility crisis.

Challenges for assessing replicability in preclinical cancer biology

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.

Is preclinical research in cancer biology reproducible enough?

The Reproducibility Project: Cancer Biology (RPCB) was established to provide evidence about reproducibility in basic and preclinical cancer research, and to identify the factors that influence reproducibility more generally. In this commentary we address some of the scientific, ethical and policy implications of the project. We liken the basic and preclinical cancer research enterprise to a vast 'diagnostic machine' that is used to determine which clinical hypotheses should be advanced for further development, including clinical trials. The results of the RPCB suggest that this diagnostic machine currently recommends advancing many findings that are not reproducible. While concerning, we believe that more work needs to be done to evaluate the performance of the diagnostic machine. Specifically, we believe three questions remain unanswered: how often does the diagnostic machine correctly recommend against advancing real effects to clinical testing?; what are the relative costs to society of false positive and false negatives?; and how well do scientists and others interpret the outputs of the machine?

What have we learned?

As the final outputs of the Reproducibility Project: Cancer Biology are published, it is clear that preclinical research in cancer biology is not as reproducible as it should be.

Is preclinical research in cancer biology reproducible enough?

The Reproducibility Project: Cancer Biology (RPCB) was established to provide evidence about reproducibility in basic and preclinical cancer research, and to identify the factors that influence reproducibility more generally. In this commentary we address some of the scientific, ethical and policy implications of the project. We liken the basic and preclinical cancer research enterprise to a vast 'diagnostic machine' that is used to determine which clinical hypotheses should be advanced for further development, including clinical trials. The results of the RPCB suggest that this diagnostic machine currently recommends advancing many findings that are not reproducible. While concerning, we believe that more work needs to be done to evaluate the performance of the diagnostic machine. Specifically, we believe three questions remain unanswered: how often does the diagnostic machine correctly recommend against advancing real effects to clinical testing?; what are the relative costs to society of false positive and false negatives?; and how well do scientists and others interpret the outputs of the machine?

Challenges for assessing replicability in preclinical cancer biology

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.

Investigating the replicability of preclinical cancer biology

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.