An investigation of new toxicity test method performance in validation studies: 1. Toxicity test methods that have predictive capacity no greater than chance

A Critical Assessment of the OECD Collaborative Study to Validate the Uterotrophic Assay for the Detection of Oestrogenic and Anti-oestrogenic Chemicals

The design and execution of a recently completed validation study on the rat uterotrophic assay for detecting oestrogens and anti-oestrogens, managed by the OECD, are critically assessed with respect to internationally agreed criteria for the validation of new in vitro and in vivo toxicity test methods. It is concluded that, while the design of the study did not take account of several important criteria for validation, the uterotrophic assay appears to reliably detect the strong and weak oestrogenic substances used in the study, which act via binding to the oestrogen receptor in vivo. However, the reliability of the assay has not been substantiated for detecting anti-oestrogens that act as antagonists, due to the involvement of an insufficient number of experiments and test chemicals. Moreover, the data do not permit an assessment of the accuracy of the prediction of oestrogenicity, and the protocols have not been sufficiently optimised with regard to controlling variables. This problem has been exacerbated by a wish to introduce as much flexibility as possible into the protocols during the formal validation phase of the study, rather than during a separate prevalidation stage. In addition, the choice between surgically treated and/or immature animals, and details of housing and husbandry conditions that are necessary for increasing the sensitivity and efficiency of the assay, need to be clarified. The assay also lacks a well-defined prediction model by which the overall relevance of the data to toxicity, and especially to human hazard, can be assessed, and no performance criteria have been established. The results of this analysis of the study indicate that it would be premature to produce an OECD test guideline for the uterotrophic assay at this time, before some of the above issues have been satisfactorily resolved.

An investigation of new toxicity test method performance in validation studies: 1. Toxicity test methods that have predictive capacity no greater than chance

An approach commonly used to measure new toxicity test method (NTM) performance in validation studies is to divide toxicity results into positive and negative classifications, and the identify true positive (TP), true negative (TN), false positive (FP) and false negative (FN) results. After this step is completed, the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are calculated. Although these statistics are widely used and often the only statistics used to assess the performance of toxicity test methods, there is little specific guidance in the validation literature on what values for these statistics indicate adequate performance. The purpose of this study was to begin developing data-based answers to this question by characterizing the CPS obtained from an NTM whose data have a completely random association with a reference test method (RTM). Determining the CPS of this worst-case scenario is useful because it provides a lower baseline from which the performance of an NTM can be judged in future validation studies. It also provides an indication of relationships in the CPS that help identify random or near-random relationships in the data. The results from this study of randomly associated tests show that the values obtained for the statistics vary significantly depending on the cut-offs chosen, that high values can be obtained for individual statistics, and that the different measures cannot be considered independently when evaluating the performance of an NTM. When the association between results of an NTM and RTM is random the sum of the complementary pairs of statistics (sensitivity + specificity, NPV + PPV) is approximately 1, and the prevalence (i.e., the proportion of toxic chemicals in the population of chemicals) and PPV are equal. Given that combinations of high sensitivity-low specificity or low specificity-high sensitivity (i.e., the sum of the sensitivity and specificity equal to approximately 1) indicate lack of predictive capacity, an NTM having these performance characteristics should be considered no better for predicting toxicity than by chance alone.

An investigation of new toxicity test method performance in validation studies: 3. Sensitivity and specificity are not independent of prevalence or distribution of toxicity

Often, the only measures of toxicity test performance provided in validation studies are the contingent probability statistics (CPS) sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Sensitivity and specificity are generally used in preference to NPV and PPV since NPV and PPV are assumed to vary with changes in prevalence while sensitivity and specificity are assumed to be independent of changes in prevalence. The purpose of the studies reported here was to test whether or not sensitivity and specificity are actually independent of changes in prevalence. Results derived from these studies indicate that sensitivity and specificity vary significantly depending on the prevalence of toxic substances in the set of chemicals being tested. This means sensitivity and specificity should not always be considered constant indicators of toxicity test performance.