Using historical control data in bioassays for regulatory toxicology

Considerations for the Identification and Conveyance of Clinical Pathology Findings in Preclinical Toxicity Studies: Results From the 9th ESTP International Expert Workshop

The European Society of Toxicologic Pathology (ESTP) organized a panel of 24 international experts from many fields of toxicologic clinical pathology (e.g., industry, academia, and regulatory) that came together in 2021 to align the use of terminology to convey the importance of clinical pathology findings in preclinical toxicity studies. An additional goal consisted of how to identify important findings in standard and nonstandard clinical pathology associated endpoints. This manuscript summarizes the information and opinions discussed and shared at the ninth ESTP International Expert Workshop, April 5 to 6, 2022. In addition to terminology usage, the workshop considered topics related to the identification and conveyance of the importance of test item-related findings. These topics included sources of variability, comparators, statistics, reporting, correlations to other study data, nonstandard biomarkers, indirect/secondary findings, and an overall weight-of-evidence approach.

Replacing concurrent controls with virtual control groups in rat toxicity studies

Virtual control groups (VCGs) in nonclinical toxicity represent the concept of using appropriate historical control data for replacing concurrent control group animals. Historical control data collected from standardized studies can serve as base for constructing VCGs and legacy study reports can be used as a benchmark to evaluate the VCG performance. Replacing concurrent controls of legacy studies with VCGs should ideally reproduce the results of these studies. Based on three four-week rat oral toxicity legacy studies with varying degrees of toxicity findings we developed a concept to evaluate VCG performance on different levels: the ability of VCGs to (i) reproduce statistically significant deviations from the concurrent control, (ii) reproduce test substance-related effects, and (iii) reproduce the conclusion of the toxicity study in terms of threshold dose, target organs, toxicological biomarkers (clinical pathology) and reversibility. Although VCGs have shown a low to moderate ability to reproduce statistical results, the general study conclusions remained unchanged. Our results provide a first indication that carefully selected historical control data can be used to replace concurrent control without impairing the general study conclusion. Additionally, the developed procedures and workflows lay the foundation for the future validation of virtual controls for a use in regulatory toxicology.

In vivo alkaline comet assay: Statistical considerations on historical negative and positive control data

The alkaline comet assay is frequently used as in vivo follow-up test within different regulatory environments to characterize the DNA-damaging potential of different test items. The corresponding OECD Test guideline 489 highlights the importance of statistical analyses and historical control data (HCD) but does not provide detailed procedures. Therefore, the working group "Statistics" of the German-speaking Society for Environmental Mutation Research (GUM) collected HCD from five laboratories and >200 comet assay studies and performed several statistical analyses. Key results included that (I) observed large inter-laboratory effects argue against the use of absolute quality thresholds, (II) > 50% zero values on a slide are considered problematic, due to their influence on slide or animal summary statistics, (III) the type of summarizing measure for single-cell data (e.g., median, arithmetic and geometric mean) may lead to extreme differences in resulting animal tail intensities and study outcome in the HCD. These summarizing values increase the reliability of analysis results by better meeting statistical model assumptions, but at the cost of information loss. Furthermore, the relation between negative and positive control groups in the data set was always satisfactorily (or sufficiently) based on ratio, difference and quantile analyses.

Historical control histopathology data from amphibian metamorphosis assays and fathead minnow fish short term reproductive assays: A tool for data interpretation

The Amphibian Metamorphosis Assay (AMA) is used to determine if a tested chemical has potential to impact the hypothalamic-pituitary-thyroid (HPT) axis of Xenopus laevis tadpoles, while the Fish Short Term Reproduction Assay (FSTRA) assesses potential effects on the hypothalamic-pituitary-gonadal (HPG) axis of fish such as the fathead minnow (Pimephales promelas). Several global regulatory programs routinely require these internationally validated tests be performed to determine the potential endocrine activity of chemicals. As such, they are conducted in accordance with standardized protocols and test criteria, which were originally developed more than a decade ago. Sizeable numbers of AMA and FSTRA studies have since been carried out, which allows for the mining of extensive historical control data (HCD). Such data are useful for investigating the existence of outlier results and aberrant control groups, identifying potential confounding variables, providing context for rare diagnoses, discriminating target from non-target effects, and for refining current testing paradigms. The present paper provides histopathology HCD from 55 AMA studies and 45 fathead minnow FSTRA studies, so that these data may become publicly available and thus aid in the interpretation of future study outcomes. Histopathology is a key endpoint in these assays, in which it is considered to be one of the most sensitive indicators of endocrine perturbation. In the current review, granular explorations of HCD data were used to identify background lesions, to assess the utility of particular diagnostic findings for distinguishing endocrine from non-endocrine effects, and to help determine if specific improvements to established regulatory guidance may be warranted. Knowledge gleaned from this investigation, supplemented by information from other recent studies, provided further context for the interpretation of AMA and FSTRA histopathology results. We recommend HCDs for the AMA and FSTRA be maintained to support the interpretation of study results.

The calculation of historical control limits in toxicology: Do's, don'ts and open issues from a statistical perspective

For reporting toxicology studies, the presentation of historical control data and the validation of the concurrent control group with respect to historical control limits have become requirements. However, many regulatory guidelines fail to define how such limits should be calculated and what kind of target value(s) they should cover. Hence, this manuscript is aimed to give a brief review on the methods for the calculation of historical control limits that are in use as well as on their theoretical background. Furthermore, this manuscript is aimed to identify open issues for the use of historical control limits that need to be discussed by the community. It seems that, even after 40 years of discussion, more issues remain open than solved, both, with regard to the available methodology as well as its implementation in user-friendly software. Since several of these topics equally apply to several research fields, this manuscript is addressed to all relevant stakeholders who deal with historical control data obtained from toxicological studies, regardless of their background or field of research.

Guidance for statistical design and analysis of toxicological dose-response experiments, based on a comprehensive literature review

The analysis of dose-response, concentration-response, and time-response relationships is a central component of toxicological research. A major decision with respect to the statistical analysis is whether to consider only the actually measured concentrations or to assume an underlying (parametric) model that allows extrapolation. Recent research suggests the application of modelling approaches for various types of toxicological assays. However, there is a discrepancy between the state of the art in statistical methodological research and published analyses in the toxicological literature. The extent of this gap is quantified in this work using an extensive literature review that considered all dose-response analyses published in three major toxicological journals in 2021. The aspects of the review include biological considerations (type of assay and of exposure), statistical design considerations (number of measured conditions, design, and sample sizes), and statistical analysis considerations (display, analysis goal, statistical testing or modelling method, and alert concentration). Based on the results of this review and the critical assessment of three selected issues in the context of statistical research, concrete guidance for planning, execution, and analysis of dose-response studies from a statistical viewpoint is proposed.

Can Historical Control Group Data Be Used to Replace Concurrent Controls in Animal Studies?

The availability of large amounts of high-quality control data from tightly controlled regulated animal safety data has created the idea to re-use these data beyond its classical applications of quality control, identification of treatment-related effects and assessing effect-size relevance for building virtual control groups (VCGs). While the ethical and cost-saving aspects of such a concept are immediately evident, the potential challenges need to be carefully considered to avoid any effect which could lower the sensitivity of an animal study to detect adverse events, safety thresholds, target organs, or biomarkers. In our brief communication, we summarize the current discussion regarding VCGs and propose a path forward how the replacement of concurrent control with VCGs resulting from historical data could be systematically assessed and to come to conclusions regarding the scientific value of the concept.

Practical guidance to evaluate in vitro dermal absorption studies for pesticide registration: An industry perspective

While there are some regulatory assessment criteria available on how to generally evaluate dermal absorption (DA) studies for risk assessment purposes, practical guidance and examples are lacking. The current manuscript highlights the challenges in interpretating data from in vitro assays and proposes holistic data-based assessment strategies from an industry perspective. Inflexible decision criteria may be inadequate for real data and may lead to irrelevant DA estimates. We recommend the use of mean values for reasonably conservative DA estimates from in vitro studies. In cases where additional conservatism is needed, e.g., due to non-robust data and acute exposure scenarios, the upper 95% confidence interval of the mean may be appropriate. It is critical to review the data for potential outliers and we provide some example cases and strategies to identify aberrant responses. Some regional regulatory authorities require the evaluation of stratum corneum (SC) residue, but here, as a very simple pro-rata approach, we propose to review whether the predicted post 24-h absorption flux exceeds the predicted elimination flux by desquamation because otherwise it is not possible for the SC residue to contribute to systemic dose. Overall, the adjustment of DA estimates due to mass balance (normalization) is not recommended.

Assessing the quality and making appropriate use of historical negative control data: A report of the International Workshop on Genotoxicity Testing (IWGT)

Historical negative control data (HCD) have played an increasingly important role in interpreting the results of genotoxicity tests. In particular, Organisation for Economic Co-operation and Development (OECD) genetic toxicology test guidelines recommend comparing responses produced by exposure to test substances with the distribution of HCD as one of three criteria for evaluating and interpreting study results (referred to herein as "Criterion C"). Because of the potential for inconsistency in how HCD are acquired, maintained, described, and used to interpret genotoxicity testing results, a workgroup of the International Workshops for Genotoxicity Testing was convened to provide recommendations on this crucial topic. The workgroup used example data sets from four in vivo tests, the Pig-a gene mutation assay, the erythrocyte-based micronucleus test, the transgenic rodent gene mutation assay, and the in vivo alkaline comet assay to illustrate how the quality of HCD can be evaluated. In addition, recommendations are offered on appropriate methods for evaluating HCD distributions. Recommendations of the workgroup are: When concurrent negative control data fulfill study acceptability criteria, they represent the most important comparator for judging whether a particular test substance induced a genotoxic effect. HCD can provide useful context for interpreting study results, but this requires supporting evidence that (i) HCD were generated appropriately, and (ii) their quality has been assessed and deemed sufficiently high for this purpose. HCD should be visualized before any study comparisons take place; graph(s) that show the degree to which HCD are stable over time are particularly useful. Qualitative and semi-quantitative assessments of HCD should also be supplemented with quantitative evaluations. Key factors in the assessment of HCD include: (i) the stability of HCD over time, and (ii) the degree to which inter-study variation explains the total variability observed. When animal-to-animal variation is the predominant source of variability, the relationship between responses in the study and an HCD-derived interval or upper bounds value (i.e., OECD Criterion C) can be used with a strong degree of confidence in contextualizing a particular study's results. When inter-study variation is the major source of variability, comparisons between study data and the HCD bounds are less useful, and consequentially, less emphasis should be placed on using HCD to contextualize a particular study's results. The workgroup findings add additional support for the use of HCD for data interpretation; but relative to most current OECD test guidelines, we recommend a more flexible application that takes into consideration HCD quality. The workgroup considered only commonly used in vivo tests, but it anticipates that the same principles will apply to other genotoxicity tests, including many in vitro tests.

Retrospective analysis of the potential use of virtual control groups in preclinical toxicity assessment using the eTOX database

Virtual Control Groups (VCGs) based on Historical Control Data (HCD) in preclinical toxicity testing have the potential to reduce animal usage. As a case study we retrospectively analyzed the impact of replacing Concurrent Control Groups (CCGs) with VCGs on the treatment-relatedness of 28 selected histopathological findings reported in either rat or dog in the eTOX database. We developed a novel methodology whereby statistical predictions of treatment-relatedness using either CCGs or VCGs of varying covariate similarity to CCGs were compared to designations from original toxicologist reports; and changes in agreement were used to quantify changes in study outcomes. Generally, the best agreement was achieved when CCGs were replaced with VCGs with the highest level of similarity; the same species, strain, sex, administration route, and vehicle. For example, balanced accuracies for rat findings were 0.704 (predictions based on CCGs) vs. 0.702 (predictions based on VCGs). Moreover, we identified covariates which resulted in poorer identification of treatment-relatedness. This was related to an increasing incidence rate divergence in HCD relative to CCGs. Future databases which collect data at the individual animal level including study details such as animal age and testing facility are required to build adequate VCGs to accurately identify treatment-related effects.

Hurdles and signposts on the road to virtual control groups-A case study illustrating the influence of anesthesia protocols on electrolyte levels in rats

Introduction: Virtual Control Groups (VCGs) represent the concept of using historical control data from legacy animal studies to replace concurrent control group (CCG) animals. Based on the data curation and sharing activities of the Innovative Medicine Initiatives project eTRANSAFE (enhancing TRANSlational SAFEty Assessment through Integrative Knowledge Management) the ViCoG working group was established with the objectives of i) collecting suitable historical control data sets from preclinical toxicity studies, ii) evaluating statistical methodologies for building adequate and regulatory acceptable VCGs from historical control data, and iii) sharing those control-group data across multiple pharmaceutical companies. During the qualification process of VCGs a particular focus was put on the identification of hidden confounders in the data sets, which might impair the adequate matching of VCGs with the CCG. Methods: During our analyses we identified such a hidden confounder, namely, the choice of the anesthetic procedure used in animal experiments before blood withdrawal. Anesthesia using CO₂ may elevate the levels of some electrolytes such as calcium in blood, while the use of isoflurane is known to lower these values. Identification of such hidden confounders is particularly important if the underlying experimental information (e.g., on the anesthetic procedure) is not routinely recorded in the standard raw data files, such as SEND (Standard for Exchange of Non-clinical Data). We therefore analyzed how the replacement of CCGs with VCGs would affect the reproducibility of treatment-related findings regarding electrolyte values (potassium, calcium, sodium, and phosphate). The analyses were performed using a legacy rat systemic toxicity study consisting of a control and three treatment groups conducted according to pertinent OECD guidelines. In the report of this study treatment-related hypercalcemia was reported. The rats in this study were anesthetized with isoflurane. Results: Replacing the CCGs with VCGs derived from studies comprising both anesthetics resulted in a shift of control electrolyte parameters. Instead of the originally reported hypercalcemia the use of VCG led to fallacious conclusions of no observed effect or hypocalcemia. Discussion: Our study highlights the importance of a rigorous statistical analysis including the detection and elimination of hidden confounders prior to the implementation of the VCG concept.

Understanding and overcoming the technical challenges in using in silico predictions in regulatory decisions of complex toxicological endpoints - A pesticide perspective for regulatory toxicologists with a focus on machine learning models

There are many challenges that must be overcome before in silico toxicity predictions are ripe for regulatory decision-making. Today, mandates in the United States of America and the European Union to avoid animal usage in toxicity testing is driving the need to consider alternative technologies, including Quantitative Structure Activity Relationship (QSAR) models, and read across approaches. However, when adopting new methods, it is critical that both new approach developers as well as regulatory users understand the strengths and challenges with these new approaches. In this paper, we identify potential sources of bias in machine learning methods specific to toxicity predictions, that may impact the overall performance of in silico models. We also discuss ways to mitigate these biases. Based on our experiences, the most prevalent sources of bias include class imbalance (differing numbers of "toxic" vs "nontoxic" compounds), limited numbers of chemicals within a particular chemistry, and biases within the studies that make up the database used for model building, as well as model evaluation biases. While this is already complex for repeated dose toxicity, in reproduction and developmental toxicity a further level of complexity is introduced by the need to evaluate effects on individual animal and litter basis (e.g., a hierarchal structure). We also discuss key considerations developers and regulators need to make when they use machine learning models to predict chemical safety. Our objective is for our paper to serve as a desk reference for model developers and regulators as they evaluate machine learning models and as they make decisions using these models.

Scientific and Regulatory Policy Committee Best Practices: Recommended ("Best") Practices for Informed (Non-blinded) Versus Masked (Blinded) Microscopic Evaluation in Animal Toxicity Studies

This article describes the Society of Toxicologic Pathology's (STP) five recommended ("best") practices for appropriate use of informed (non-blinded) versus masked (blinded) microscopic evaluation in animal toxicity studies intended for regulatory review. (1) Informed microscopic evaluation is the default approach for animal toxicity studies. (2) Masked microscopic evaluation has merit for confirming preliminary diagnoses for target organs and/or defining thresholds ("no observed adverse effect level" and similar values) identified during an initial informed evaluation, addressing focused hypotheses, or satisfying guidance or requests from regulatory agencies. (3) If used as the approach for an animal toxicity study to investigate a specific research question, masking of the initial microscopic evaluation should be limited to withholding only information about the group (control or test article-treated) and dose equivalents. (4) The decision regarding whether or not to perform a masked microscopic evaluation is best made by a toxicologic pathologist with relevant experience. (5) Pathology peer review, performed to verify the microscopic diagnoses and interpretations by the study pathologist, should use an informed evaluation approach. The STP maintains that implementing these five best practices has and will continue to consistently deliver robust microscopic data with high sensitivity for animal toxicity studies intended for regulatory review. Consequently, when conducting animal toxicity studies, the advantages of informed microscopic evaluation for maximizing sensitivity outweigh the perceived advantages of minimizing bias through masked microscopic examination.

Using R in Regulatory Toxicology

Statistical analyses are an essential part of regulatory toxicological evaluations. While projects would be ideally monitored by both toxicologists and statisticians, this is often not possible in practice. Hence, toxicologists should be trained in some common statistical approaches but also need a tool for statistical evaluations. Due to transparency needed in regulatory processes and standard tests that can be evaluated with template approaches, the freely available open-source statistical software R may be suitable. R is a well-established software in the statistical community. The principal input method is via software code, which is both benefit and weakness of the tool. It is increasingly used by regulating authorities globally and can be easily extended by software packages, e.g., for new statistical functions and features. This manuscript outlines how R can be used in regulatory toxicology, allowing toxicologists to perform all regulatory required data evaluations in a single software solution. Practical applications are shown in case studies on simulated and experimental data. The examples cover a) Dunnett testing of treatment groups against a common control and in relation to a biological relevance threshold, assessing the test's assumptions and plotting the results; b) dose-response analysis and benchmark dose derivation for chronic kidney inflammation as a function of Pyridine; and c) graphical/exploratory data analysis of previously published developmental neurotoxicity data for Chlorpyrifos.

Benchmark dose modelling in regulatory ecotoxicology, a potential tool in pest management

For several authorities, benchmark dose (BMD) methodology has become the recommended approach by which to derive reference values for risk assessment. However, in practice, the BMD approach is not standard use in risk assessment for pesticides where the no observed adverse effect level, lowest observed adverse effect level and effective dose (ED₅₀ or ED_x ) prevail. Regression-based BMD and the benchmark dose lower confidence limit (BMDL) have several advantages, such as utilizing more information from the generated data and being less dependent on tested dose levels. However, the BMD approach requires some degree of expert knowledge for defining an appropriate risk level for estimating the BMD and using more sophisticated statistical methods to calculate BMD and BMDL. The BMD approach is one way to move away from p value-based binary decision-making towards putting the weight on effect sizes. We review the advantages and disadvantages of focusing on the BMD approach for risk assessment of pesticides. Further, we discuss potential applications in efficacy trials for pest management purposes. © 2021 Society of Chemical Industry.