Towards a Fuzzy Expert System on Toxicological Data Quality Assessment

How effective is score-based data quality assessment? An illustration with fish BCF data

Increasingly rigorous data quality (DQ) evaluations and/or screening practices are being applied to environmental and ecotoxicological datasets. DQ is predominantly evaluated by scoring given data against preselected criteria. This study provides the first examination on the effectiveness of score-based DQ evaluation in providing statistically meaningful differentiation of measurements using fish bioconcentration factor (BCF) dataset as an illustration. This is achieved by inspecting how log BCF differs with the built-in overall-DQ and specific-DQ evaluations, and how it is influenced by interactive effects and hierarchy of DQ criteria. Approximately 80-90% of analyzable chemicals show no statistical difference in log BCF between low-quality (LQ) and high-quality (HQ) measurements in overall evaluation (n = 183) or in individual evaluation of 6 DQ criteria (n = 53 to 101). Further examination shows that log BCF may/may not change with different combinations or total number of criteria violations. Tree analysis and nodal structures of deviation in log BCF also reveal the absence of common structural dependence on the criteria violated. Finally, simple averaging of all measurements without DQ differentiation yields comparable log BCFs as those derived using strictly HQ data with ≤0.5 log unit difference in over 93% of the chemicals (n = 158) and no dependence on number of measurements, fraction of LQ measurements, or bioaccumulation potential of the chemicals. For accurate log BCF, DQ appears no more important than having more independent measurements irrespective of their individual DQ statuses. This work concludes by calling for: (i) re-documentation of experimental details in legacy environmental and ecotoxicological datasets, (ii) examination of other DQ-categorized datasets using the tests and tools applied here, and (ii) a thorough and systematic reflection on how DQ should be assessed for modeling, benchmarking, and other data-based analyses or applications.

Development of the SciRAP Approach for Evaluating the Reliability and Relevance of in vitro Toxicity Data

Efficient and successful integration of data generated from non-animal test methods must rely on reliable and relevant data. It is important therefore to develop tools and criteria that facilitate scientifically sound, structured, and transparent evaluation of reliability and relevance of in vitro toxicity data to efficiently inform regulatory hazard and risk assessment. The Science in Risk Assessment and Policy (SciRAP) initiative aims to promote such overarching goals. We present the work to develop and refine the SciRAP tool for evaluation of reliability and relevance of in vitro studies for incorporation on the SciRAP web-based platform (www.scirap.org). In the SciRAP approach, reliability evaluation is based on criteria for reporting quality and methodological quality, and is explicitly separated from relevance evaluation. The SciRAP in vitro tool (version 1.0) was tested and evaluated during an expert test round (April 2019-September 2020) on three in vitro studies by thirty-one experts from regulatory authorities, industry and academia from different geographical areas and with various degree of experience in in vitro research and/or human health risk assessment. In addition, the experts answered an online survey to collect their feedback about the general features and desired characteristics of the tool for further refinement. The SciRAP in vitro tool (version 2.0) was revised based on the outcome of the expert test round (study evaluation and online survey) and consists of 24 criteria for evaluating "reporting quality" (reliability), 16 criteria for "methodological quality" (reliability), and 4 items for evaluating relevance of in vitro studies. Participants were generally positive about the adequacy, flexibility, and user-friendliness of the tool. The expert test round outlined the need to (i) revise the formulation of certain criteria; (ii) provide new or revised accompanying guidance for reporting quality and methodological quality criteria in the "test compounds and controls," "test system," and "data collection and analysis" domains; and (iii) provide revised guidance for relevance items, as general measures to reduce inter-expert variability. The SciRAP in vitro tool allows for a structured and transparent evaluation of in vitro studies for use in regulatory hazard and risk assessment of chemicals.

Quantifying Uncertainty in Ecotoxicological Risk Assessment: MUST, a Modular Uncertainty Scoring Tool

Whether conducting a risk, hazard, or alternatives assessment, one invariably struggles with the task of reconciling multiple available values of toxicological thresholds into a single outcome. When combining multiple pieces of evidence from many different sources, it is important to consider the role of data uncertainty. Uncertainty is inherent to all scientific data. However, in toxicological assessments, controversies and uncertainties are typically understated; they lack methodological transparency; or they poorly integrate qualitative and quantitative sources of information. Similarly, in model development, data curation is rarely performed with sufficient rigor, particularly when applying big data statistics. To overcome the hurdles of a decision process that must reconcile divergent data, we developed an uncertainty scoring tool that can be trained to reproduce specific decision-making paradigms and ensure consistency in the practitioner's judgment across complex scenarios. While designed to aid with ecotoxicological assessments and predictive model development, the tool's applicability extends to any decision-making process that calls for synthesis of incongruent data. Here, we highlight the development process, as well as demonstrate the method's utility in several prototypical ecotoxicological case studies.

Going All In: A Strategic Investment in In Silico Toxicology

As vast numbers of new chemicals are introduced to market annually, we are faced with the grand challenge of protecting humans and the environment while minimizing economically and ethically costly animal testing. In silico models promise to be the solution we seek, but we find ourselves at crossroads of future development efforts that would ensure standalone applicability and reliability of these tools. A conscientious effort that prioritizes experimental testing to support the needs of in silico models (versus regulatory needs) is called for to achieve this goal. Using economic analogy in the title of this work, we argue that a prudent investment is to go all-in to support in silico model development, rather than gamble our future by keeping the status quo of a "balanced portfolio" of testing approaches. We discuss two paths to future in silico toxicology-one based on big-data statistics ("broadsword"), and the other based on direct modeling of molecular interactions ("scalpel")-and offer rationale that the latter approach is more transparent, is better aligned with our quest for fundamental knowledge, and has a greater potential to succeed if we are willing to transform our toxicity-testing paradigm.

Requirements for Data Quality Metrics

Data quality and especially the assessment of data quality have been intensively discussed in research and practice alike. To support an economically oriented management of data quality and decision making under uncertainty, it is essential to assess the data quality level by means of well-founded metrics. However, if not adequately defined, these metrics can lead to wrong decisions and economic losses. Therefore, based on a decision-oriented framework, we present a set of five requirements for data quality metrics. These requirements are relevant for a metric that aims to support an economically oriented management of data quality and decision making under uncertainty. We further demonstrate the applicability and efficacy of these requirements by evaluating five data quality metrics for different data quality dimensions. Moreover, we discuss practical implications when applying the presented requirements.

A fuzzy rule-based expert system for diagnosing cystic fibrosis

Background

Finding a valid diagnosis is mostly a prolonged process. Current advances in the sector of artificial intelligence have led to the appearance of expert systems that enrich the experiences and capabilities of doctors for making decisions for their patients.

Objective

The objective of this research was developing a fuzzy expert system for diagnosing Cystic Fibrosis (CF).

Methods

Defining the risk factors and then, designing the fuzzy expert system for diagnosis of CF were carried out in this cross-sectional study. To evaluate the performance of the proposed system, a dataset that corresponded to 70 patients with respiratory disease who were serially admitted to the CF Clinic in the Pediatric Respiratory Diseases Center, Masih Daneshvari Hospital in Tehran, Iran during August 2016 to January 2017 was considered. Whole procedures of system construction were implemented in a MATLAB environment.

Results

Results showed that the suggested system can be used as a strong diagnostic tool with 93.02% precision, 89.29% specificity, 95.24% sensitivity and 92.86% accuracy for diagnosing CF. There was also a good relationship between the user and the system through the appealing user interface.

Conclusion

The system is equipped with information, knowledge, and expertise from certified specialists; hence, as a training tool it can be useful for new physicians. It is worth mentioning that the accomplishment of this project depends on advocacy of decision making in CF diagnosis. Nevertheless, it is expected that the system will reduce the number of false positives and false negatives in unusual cases.

A critical review of frameworks used for evaluating reliability and relevance of (eco)toxicity data: Perspectives for an integrated eco-human decision-making framework

Considerable efforts have been invested so far to evaluate and rank the quality and relevance of (eco)toxicity data for their use in regulatory risk assessment to assess chemical hazards. Many frameworks have been developed to improve robustness and transparency in the evaluation of reliability and relevance of individual tests, but these frameworks typically focus on either environmental risk assessment (ERA) or human health risk assessment (HHRA), and there is little cross talk between them. There is a need to develop a common approach that would support a more consistent, transparent and robust evaluation and weighting of the evidence across ERA and HHRA. This paper explores the applicability of existing Data Quality Assessment (DQA) frameworks for integrating environmental toxicity hazard data into human health assessments and vice versa. We performed a comparative analysis of the strengths and weaknesses of eleven frameworks for evaluating reliability and/or relevance of toxicity and ecotoxicity hazard data. We found that a frequent shortcoming is the lack of a clear separation between reliability and relevance criteria. A further gaps and needs analysis revealed that none of the reviewed frameworks satisfy the needs of a common eco-human DQA system. Based on our analysis, some key characteristics, perspectives and recommendations are identified and discussed for building a common DQA system as part of a future integrated eco-human decision-making framework. This work lays the basis for developing a common DQA system to support the further development and promotion of Integrated Risk Assessment.

How should the completeness and quality of curated nanomaterial data be evaluated?

Nanotechnology is of increasing significance. Curation of nanomaterial data into electronic databases offers opportunities to better understand and predict nanomaterials' behaviour. This supports innovation in, and regulation of, nanotechnology. It is commonly understood that curated data need to be sufficiently complete and of sufficient quality to serve their intended purpose. However, assessing data completeness and quality is non-trivial in general and is arguably especially difficult in the nanoscience area, given its highly multidisciplinary nature. The current article, part of the Nanomaterial Data Curation Initiative series, addresses how to assess the completeness and quality of (curated) nanomaterial data. In order to address this key challenge, a variety of related issues are discussed: the meaning and importance of data completeness and quality, existing approaches to their assessment and the key challenges associated with evaluating the completeness and quality of curated nanomaterial data. Considerations which are specific to the nanoscience area and lessons which can be learned from other relevant scientific disciplines are considered. Hence, the scope of this discussion ranges from physicochemical characterisation requirements for nanomaterials and interference of nanomaterials with nanotoxicology assays to broader issues such as minimum information checklists, toxicology data quality schemes and computational approaches that facilitate evaluation of the completeness and quality of (curated) data. This discussion is informed by a literature review and a survey of key nanomaterial data curation stakeholders. Finally, drawing upon this discussion, recommendations are presented concerning the central question: how should the completeness and quality of curated nanomaterial data be evaluated?

Methods for assigning confidence to toxicity data with multiple values--Identifying experimental outliers

The assessment of data quality is a crucial element in many disciplines such as predictive toxicology and risk assessment. Currently, the reliability of toxicity data is assessed on the basis of testing information alone (adherence to Good Laboratory Practice (GLP), detailed testing protocols, etc.). Common practice is to take one toxicity data point per compound - usually the one with the apparently highest reliability. All other toxicity data points (for the same experiment and compound) from other sources are neglected. To show the benefits of incorporating the "less reliable" data, a simple, independent, statistical approach to assess data quality and reliability on a mathematical basis was developed. A large data set of toxicity values to Aliivibrio fischeri was assessed. The data set contained 1813 data points for 1227 different compounds, including 203 identified as non-polar narcotic. Log KOW values were calculated and non-polar narcosis quantitative structure-activity relationship (QSAR) models were built. A statistical approach to data quality assessment, which is based on data outlier omission and confidence scoring, improved the linear QSARs. The results indicate that a beneficial method for using large data sets containing multiple data values per compound and highly variable study data has been developed. Furthermore this statistical approach can help to develop novel QSARs and support risk assessment by obtaining more reliable values for biological endpoints.

Sources of Chemical Information, Toxicity Data and Assessment of Their Quality

This chapter identifies a range of sources that provide toxicity data that may be of use in category formation and read­across. Data in this context relate to both the chemical identity and characteristics of molecules in addition to biological (toxicological) information. Different methods of representing chemicals are given and caveats associated with the use of certain representations are also indicated. A glossary of key terms relating to assessment of data quality is provided along with guidance on methods to perform data quality assessment.