Evaluation of QSAR models for predicting mutagenicity: outcome of the Second Ames/QSAR international challenge project

Quantitative structure-activity relationship (QSAR) models are powerful in silico tools for predicting the mutagenicity of unstable compounds, impurities and metabolites that are difficult to examine using the Ames test. Ideally, Ames/QSAR models for regulatory use should demonstrate high sensitivity, low false-negative rate and wide coverage of chemical space. To promote superior model development, the Division of Genetics and Mutagenesis, National Institute of Health Sciences, Japan (DGM/NIHS), conducted the Second Ames/QSAR International Challenge Project (2020-2022) as a successor to the First Project (2014-2017), with 21 teams from 11 countries participating. The DGM/NIHS provided a curated training dataset of approximately 12,000 chemicals and a trial dataset of approximately 1,600 chemicals, and each participating team predicted the Ames mutagenicity of each trial chemical using various Ames/QSAR models. The DGM/NIHS then provided the Ames test results for trial chemicals to assist in model improvement. Although overall model performance on the Second Project was not superior to that on the First, models from the eight teams participating in both projects achieved higher sensitivity than models from teams participating in only the Second Project. Thus, these evaluations have facilitated the development of QSAR models.

QSAR Methods

This chapter introduces the basis of computational chemistry and discusses how computational methods have been extended from physical to biological properties, and toxicology in particular, modeling. Since about three decades, chemical experimentation is more and more replaced by modeling and virtual experimentation, using a large core of mathematics, chemistry, physics, and algorithms. Animal and wet experiments, aimed at providing a standardized result about a biological property, can be mimicked by modeling methods, globally called in silico methods, all characterized by deducing properties starting from the chemical structures. Two main streams of such models are available: models that consider the whole molecular structure to predict a value, namely QSAR (quantitative structure-activity relationships), and models that check relevant substructures to predict a class, namely SAR. The term in silico discovery is applied to chemical design, to computational toxicology, and to drug discovery. Virtual experiments confirm hypotheses, provide data for regulation, and help in designing new chemicals.

QSAR modeling without descriptors using graph convolutional neural networks: the case of mutagenicity prediction

Deep neural networks are effective in learning directly from low-level encoded data without the need of feature extraction. This paper shows how QSAR models can be constructed from 2D molecular graphs without computing chemical descriptors. Two graph convolutional neural network-based models are presented with and without a Bayesian estimation of the prediction uncertainty. The property under investigation is mutagenicity: Models developed here predict the output of the Ames test. These models take the SMILES representation of the molecules as input to produce molecular graphs in terms of adjacency matrices and subsequently use attention mechanisms to weight the role of their subgraphs in producing the output. The results positively compare with current state-of-the-art models. Furthermore, our proposed model interpretation can be enhanced by the automatic extraction of the substructures most important in driving the prediction, as well as by uncertainty estimations.

Integrating in silico models and read-across methods for predicting toxicity of chemicals: A step-wise strategy

In silico methods and models are increasingly used for predicting properties of chemicals for hazard identification and hazard characterisation in the absence of experimental toxicity data. Many in silico models are available and can be used individually or in an integrated fashion. Whilst such models offer major benefits to toxicologists, risk assessors and the global scientific community, the lack of a consistent framework for the integration of in silico results can lead to uncertainty and even contradictions across models and users, even for the same chemicals. In this context, a range of methods for integrating in silico results have been proposed on a statistical or case-specific basis. Read-across constitutes another strategy for deriving reference points or points of departure for hazard characterisation of untested chemicals, from the available experimental data for structurally-similar compounds, mostly using expert judgment. Recently a number of software systems have been developed to support experts in this task providing a formalised and structured procedure. Such a procedure could also facilitate further integration of the results generated from in silico models and read-across. This article discusses a framework on weight of evidence published by EFSA to identify the stepwise approach for systematic integration of results or values obtained from these "non-testing methods". Key criteria and best practices for selecting and evaluating individual in silico models are also described, together with the means to combining the results, taking into account any limitations, and identifying strategies that are likely to provide consistent results.

Could deep learning in neural networks improve the QSAR models?

Assessing chemical toxicity is a multidisciplinary process, traditionally involving in vivo, in vitro and in silico tests. Currently, toxicological goal is to reduce new tests on chemicals, exploiting all information yet available. Recent advancements in machine learning and deep neural networks allow computers to automatically mine patterns and learn from data. This technology, applied to (Q)SAR model development, leads to discover by learning the structural-chemical-biological relationships and the emergent properties. Starting from Toxception, a deep neural network predicting activity from the chemical graph image, we designed SmilesNet, a recurrent neural network taking SMILES as the only input. We then integrated the two networks into C-Tox network to make the final classification. Results of our networks, trained on a ~20K molecule dataset with Ames test experimental values, match or even outperform the current state of the art. We also extract knowledge from the networks and compare it with the available mutagenic structural alerts. The advantage over traditional QSAR modelling is that our models automatically extract the features without using descriptors. Nevertheless, the model is successful if large numbers of examples are provided and computation is more complex than in classical methods.

A large comparison of integrated SAR/QSAR models of the Ames test for mutagenicity$

Results from the Ames test are the first outcome considered to assess the possible mutagenicity of substances. Many QSAR models and structural alerts are available to predict this endpoint. From a regulatory point of view, the recommendation from international authorities is to consider the predictions of more than one model and to combine results in order to develop conclusions about the mutagenicity risk posed by chemicals. However, the results of those models are often conflicting, and the existing inconsistency in the predictions requires intelligent strategies to integrate them. In our study, we evaluated different strategies for combining results of models for Ames mutagenicity, starting from a set of 10 diverse individual models, each built on a dataset of around 6000 compounds. The novelty of our study is that we collected a much larger set of about 18,000 compounds and used the new data to build a family of integrated models. These integrations used probabilistic approaches, decision theory, machine learning, and voting strategies in the integration scheme. Results are discussed considering balanced or conservative perspectives, regarding the possible uses for different purposes, including screening of large collection of substances for prioritization.

QSAR: What Else?

QSAR (quantitative structure-activity relationship) is a method for predicting the physical and biological properties of small molecules; it is today in large use in companies and public services. However, as any scientific method, it is nowadays challenged by more and more requests, especially considering its possible role in assessing the safety of new chemicals. Posing the question whether QSAR is a way not only to exploit available knowledge but also to build new knowledge, we shortly review QSAR history, thus searching for a QSAR epistemology. We consider the three pillars on which QSAR stands: biological data, chemical knowledge, and modeling algorithms. Most of the time we assume that biological data is a true picture of the world (as they result from good experimental practice), that chemical knowledge is scientifically true; so if a QSAR is not working, blame modeling. This opens the way to look at the role of modeling in developing scientific theories, and in producing knowledge. QSAR is a mature technology; however, debate is still active in many topics, in particular about the acceptability of the models and how they are explained. After an excursus in inductive reasoning, we relate the QSAR methodology to open debates in the philosophy of science.

Processing of surface EMG through pattern recognition techniques aimed at classifying shoulder joint movements

Artificial arms for shoulder disarticulation need a high number of degrees of freedom to be controlled. In order to control a prosthetic shoulder joint, an intention detection system based on surface electromyography (sEMG) pattern recognition methods was proposed and experimentally investigated. Signals from eight trunk muscles that are generally preserved after shoulder disarticulation were recorded from a group of eight normal subjects in nine shoulder positions. After data segmentation, four different features were extracted (sample entropy, cepstral coefficients of the 4th order, root mean square and waveform length) and classified by means of linear discriminant analysis. The classification accuracy was 92.1% and this performance reached 97.9% after reducing the positions considered to five classes. To reduce the computational cost, the two channels with the least discriminating information were neglected yielding to a classification accuracy diminished by just 4.08%.

Results of a round-robin exercise on read-across

A round-robin exercise was conducted within the CALEIDOS LIFE project. The participants were invited to assess the hazard posed by a substance, applying in silico methods and read-across approaches. The exercise was based on three endpoints: mutagenicity, bioconcentration factor and fish acute toxicity. Nine chemicals were assigned for each endpoint and the participants were invited to complete a specific questionnaire communicating their conclusions. The interesting aspect of this exercise is the justification behind the answers more than the final prediction in itself. Which tools were used? How did the approach selected affect the final answer?

New clues on carcinogenicity-related substructures derived from mining two large datasets of chemical compounds

In this study, new molecular fragments associated with genotoxic and nongenotoxic carcinogens are introduced to estimate the carcinogenic potential of compounds. Two rule-based carcinogenesis models were developed with the aid of SARpy: model R (from rodents' experimental data) and model E (from human carcinogenicity data). Structural alert extraction method of SARpy uses a completely automated and unbiased manner with statistical significance. The carcinogenicity models developed in this study are collections of carcinogenic potential fragments that were extracted from two carcinogenicity databases: the ANTARES carcinogenicity dataset with information from bioassay on rats and the combination of ISSCAN and CGX datasets, which take into accounts human-based assessment. The performance of these two models was evaluated in terms of cross-validation and external validation using a 258 compound case study dataset. Combining R and H predictions and scoring a positive or negative result when both models are concordant on a prediction, increased accuracy to 72% and specificity to 79% on the external test set. The carcinogenic fragments present in the two models were compared and analyzed from the point of view of chemical class. The results of this study show that the developed rule sets will be a useful tool to identify some new structural alerts of carcinogenicity and provide effective information on the molecular structures of carcinogenic chemicals.

Integrating QSAR and read-across for environmental assessment

Read-across and QSAR have different traditions and drawbacks. We address here two main questions: (1) How do we solve the issue of the subjectivity in the evaluation of data and results, which may be particularly critical for read-across, but may have a role also for the QSAR assessment? (2) How do we take advantage of the results of both approaches to support each other? The QSAR model starts from the training set. The presence of similar chemicals with property values close to that predicted can support the result. The approach in read-across is the opposite. The assessment is focused on the few substances similar to the target. The data quality of the similar chemicals is fundamental. A risk is poor standardization in the definition of 'similarity', because different approaches may be applied. Inspired by the principles of high transparency and reproducibility, a new program for read-across, called ToxRead, has been developed and made freely available ( www.toxgate.eu ). The output of ToxRead can be compared and integrated with the output of QSAR, within a weight-of-evidence strategy. We discuss the evaluation and integration of ToxRead and QSAR with examples of the assessment of bioconcentration factors of chemicals.

ToxRead: a tool to assist in read across and its use to assess mutagenicity of chemicals

Life sciences, and toxicology in particular, are heavily impacted by the development of methods for data collection and data analysis; they are moving from an analytical approach to a modelling approach. The scarce availability of experimental data is a known bottleneck in assessing the properties of new chemicals. Even when a model is available, the resulting predictions have to be assessed by close scrutiny of the chemicals and the biological properties of the compounds concerned. To avoid unnecessary testing, a read across strategy is often suggested and used. In this paper we discuss how to improve and standardize read across activity using ad hoc visualization and data search methods which use similarity measures and fragment search to organize in a chart a picture of all the relevant information that the expert needs to make an assessment. We show in particular how to apply our system to the case of mutagenicity.

A new in silico classification model for ready biodegradability, based on molecular fragments

Regulations such as the European REACH (Registration, Evaluation, Authorization and restriction of Chemicals) often require chemicals to be evaluated for ready biodegradability, to assess the potential risk for environmental and human health. Because not all chemicals can be tested, there is an increasing demand for tools for quick and inexpensive biodegradability screening, such as computer-based (in silico) theoretical models. We developed an in silico model starting from a dataset of 728 chemicals with ready biodegradability data (MITI-test Ministry of International Trade and Industry). We used the novel software SARpy to automatically extract, through a structural fragmentation process, a set of substructures statistically related to ready biodegradability. Then, we analysed these substructures in order to build some general rules. The model consists of a rule-set made up of the combination of the statistically relevant fragments and of the expert-based rules. The model gives good statistical performance with 92%, 82% and 76% accuracy on the training, test and external set respectively. These results are comparable with other in silico models like BIOWIN developed by the United States Environmental Protection Agency (EPA); moreover this new model includes an easily understandable explanation.

CORAL: binary classifications (active/inactive) for Liver-Related Adverse Effects of Drugs

Classification data related to the Liver-Related Adverse Effects of Drugs have been studied with the CORAL software (http://www.insilico.eu/coral). Two datasets which contain compounds with two serum enzyme markers of liver toxicity: alanine aminotransferase (ALT, n=187) and aspartate aminotransferase (AST, n=209) are analyzed. Statistical quality of the prediction for ALT activity is n=35, Sensitivity = 0.5556, Specificity = 0.8077, and Accuracy = 0.7429. In the case of AST activity the prediction is characterized by n=42, Sensitivity = 0.6875, Specificity = 0.7692, and Accuracy = 0.7381. A number of structural alerts which can be related to the studied activities are revealed. It is the first attempt to build up the classification QSAR model by means of the Monte Carlo technique based on representation of the molecular structure by SMILES using the CORAL software.

QSAR as a random event: modeling of nanoparticles uptake in PaCa2 cancer cells

Quantitative structure-property/activity relationships (QSPRs/QSARs) are a tool to predict various endpoints for various substances. The "classic" QSPR/QSAR analysis is based on the representation of the molecular structure by the molecular graph. However, simplified molecular input-line entry system (SMILES) gradually becomes most popular representation of the molecular structure in the databases available on the Internet. Under such circumstances, the development of molecular descriptors calculated directly from SMILES becomes attractive alternative to "classic" descriptors. The CORAL software (http://www.insilico.eu/coral) is provider of SMILES-based optimal molecular descriptors which are aimed to correlate with various endpoints. We analyzed data set on nanoparticles uptake in PaCa2 pancreatic cancer cells. The data set includes 109 nanoparticles with the same core but different surface modifiers (small organic molecules). The concept of a QSAR as a random event is suggested in opposition to "classic" QSARs which are based on the only one distribution of available data into the training and the validation sets. In other words, five random splits into the "visible" training set and the "invisible" validation set were examined. The SMILES-based optimal descriptors (obtained by the Monte Carlo technique) for these splits are calculated with the CORAL software. The statistical quality of all these models is good.

QSAR models for inhibitors of physiological impact of Escherichia coli that leads to diarrhea

Quantitative structure - activity relationships (QSARs) developed to evaluate percentage of inhibition of STa-stimulated (Escherichia coli) cGMP accumulation in T84 cells are calculated by the Monte Carlo method. This endpoint represents a measure of biological activity of a substance against diarrhea. Statistical quality of the developed models is quite good. The approach is tested using three random splits of data into the training and test sets. The statistical characteristics for three splits are the following: (1) n=20, r(2)=0.7208, q(2)=0.6583, s=16.9, F=46 (training set); n=11, r(2)=0.8986, s=14.6 (test set); (2) n=19, r(2)=0.6689, q(2)=0.5683, s=17.6, F=34 (training set); n=12, r(2)=0.8998, s=12.1 (test set); and (3) n=20, r(2)=0.7141, q(2)=0.6525, s=14.7, F=45 (training set); n=11, r(2)=0.8858, s=19.5 (test set). Based on the proposed here models hypothetical compounds which can be useful agents against diarrhea are suggested.

Automatic knowledge extraction from chemical structures: the case of mutagenicity prediction

This work proposes a new structure-activity relationship (SAR) approach to mine molecular fragments that act as structural alerts for biological activity. The entire process is designed to fit with human reasoning, not only to make the predictions more reliable but also to permit clear control by the user in order to meet customized requirements. This approach has been tested on the mutagenicity endpoint, showing marked prediction skills and, more interestingly, bringing to the surface much of the knowledge already collected in the literature as well as new evidence.

CORAL: QSPR model of water solubility based on local and global SMILES attributes

Water solubility is an important characteristic of a chemical in many aspects. However experimental definition of the endpoint for all substances is impossible. In this study quantitative structure-property relationships (QSPRs) for negative logarithm of water solubility-logS (mol L(-1)) are built up for five random splits into the sub-training set (≈55%), the calibration set (≈25%), and the test set (≈20%). Simplified molecular input-line entry system (SMILES) is used as the representation of the molecular structure. Optimal SMILES-based descriptors are calculated by means of the Monte Carlo method using the CORAL software (http://www.insilico.eu/coral). These one-variable models for water solubility are characterized by the following average values of the statistical characteristics: n(sub_train)=725-763; n(calib)=312-343; n(test)=231-261; r(sub_train)(2)=0.9211±0.0028; r(calib)(2)=0.9555±0.0045; r(test)(2)=0.9365±0.0073; s(sub_train)=0.561±0.0086; s(calib)=0.453±0.0209; s(test)=0.520±0.0205. Thus, the reproducibility of statistical quality of suggested models for water solubility confirmed for five various splits.

Novel application of the CORAL software to model cytotoxicity of metal oxide nanoparticles to bacteria Escherichia coli

Convenient to apply and available on the Internet software CORAL (http://www.insilico.eu/CORAL) has been used to build up quantitative structure-activity relationships (QSAR) for prediction of cytotoxicity of metal oxide nanoparticles to bacteria Escherichia coli (minus logarithm of concentration for 50% effect pEC50). In this study six random splits of the data into the training and test set were examined. It has been shown that the CORAL provides a reliable tool that could be used to build up a QSAR of the pEC50.