PERform: assessing model performance with predictivity and explainability readiness formula

In the rapidly evolving field of artificial intelligence (AI), explainability has been traditionally assessed in a post-modeling process and is often subjective. In contrary, many quantitative metrics have been routinely used to assess a model's performance. We proposed a unified formular named PERForm, by incorporating explainability as a weight into the existing statistical metrics to provide an integrated and quantitative measure of both predictivity and explainability to guide model selection, application, and evaluation. PERForm was designed as a generic formula and can be applied to any data types. We applied PERForm on a range of diverse datasets, including DILIst, Tox21, and three MAQC-II benchmark datasets, using various modeling algorithms to predict a total of 73 distinct endpoints. For example, AdaBoost algorithms exhibited superior performance (PERForm AUC for AdaBoost is 0.129 where Linear regression is 0) in DILIst prediction, where linear regression outperformed other models in the majority of Tox21 endpoints (PERForm AUC for linear regression is 0.301 where AdaBoost is 0.283 in average). This research marks a significant step toward comprehensively evaluating the utility of an AI model to advance transparency and interpretability, where the tradeoff between a model's performance and its interpretability can have profound implications.

Towards accurate indel calling for oncopanel sequencing through an international pipeline competition at precisionFDA

Accurately calling indels with next-generation sequencing (NGS) data is critical for clinical application. The precisionFDA team collaborated with the U.S. Food and Drug Administration's (FDA's) National Center for Toxicological Research (NCTR) and successfully completed the NCTR Indel Calling from Oncopanel Sequencing Data Challenge, to evaluate the performance of indel calling pipelines. Top performers were selected based on precision, recall, and F1-score. The performance of many other pipelines was close to the top performers, which produced a top cluster of performers. The performance was significantly higher in high confidence regions and coding regions, and significantly lower in low complexity regions. Oncopanel capture and other issues may have occurred that affected the recall rate. Indels with higher variant allele frequency (VAF) may generally be called with higher confidence. Many of the indel calling pipelines had good performance. Some of them performed generally well across all three oncopanels, while others were better for a specific oncopanel. The performance of indel calling can further be improved by restricting the calls within high confidence intervals (HCIs) and coding regions, and by excluding low complexity regions (LCR) regions. Certain VAF cut-offs could be applied according to the applications.

A framework enabling LLMs into regulatory environment for transparency and trustworthiness and its application to drug labeling document

Regulatory agencies consistently deal with extensive document reviews, ranging from product submissions to both internal and external communications. Large Language Models (LLMs) like ChatGPT can be invaluable tools for these tasks, however present several challenges, particularly the proprietary information, combining customized function with specific review needs, and transparency and explainability of the model's output. Hence, a localized and customized solution is imperative. To tackle these challenges, we formulated a framework named askFDALabel on FDA drug labeling documents that is a crucial resource in the FDA drug review process. AskFDALabel operates within a secure IT environment and comprises two key modules: a semantic search and a Q&A/text-generation module. The Module S built on word embeddings to enable comprehensive semantic queries within labeling documents. The Module T utilizes a tuned LLM to generate responses based on references from Module S. As the result, our framework enabled small LLMs to perform comparably to ChatGPT with as a computationally inexpensive solution for regulatory application. To conclude, through AskFDALabel, we have showcased a pathway that harnesses LLMs to support agency operations within a secure environment, offering tailored functions for the needs of regulatory research.

Generation of a drug-induced renal injury list to facilitate the development of new approach methodologies for nephrotoxicity

Drug-induced renal injury (DIRI) causes >1.5 million adverse events annually in the USA alone. Although standard biomarkers exist for DIRI, they lack the sensitivity or specificity to detect nephrotoxicity before the significant loss of renal function. In this study, we describe the creation of DIRIL - a list of drugs associated with DIRI and nephrotoxicity - from two literature datasets with DIRI annotation, confirmed using FDA drug labeling. DIRIL comprises 317 orally administered drugs covering all 14 anatomical, therapeutic and chemical (ATC) classification categories. Of the 317 drugs, 171 were DIRI-positive and 146 were DIRI-negative. DIRIL will be a relevant and invaluable resource for discovery of new approach methods (NAMs) to predict the occurrence and possible severity of DIRI earlier in drug development.

Measurement and Mitigation of Bias in Artificial Intelligence: A Narrative Literature Review for Regulatory Science

Artificial intelligence (AI) is increasingly being used in decision making across various industries, including the public health arena. Bias in any decision-making process can significantly skew outcomes, and AI systems have been shown to exhibit biases at times. The potential for AI systems to perpetuate and even amplify biases is a growing concern. Bias, as used in this paper, refers to the tendency toward a particular characteristic or behavior, and thus, a biased AI system is one that shows biased associations entities. In this literature review, we examine the current state of research on AI bias, including its sources, as well as the methods for measuring, benchmarking, and mitigating it. We also examine the biases and methods of mitigation specifically relevant to the healthcare field and offer a perspective on bias measurement and mitigation in regulatory science decision making.

Extend the benchmarking indel set by manual review using the individual cell line sequencing data from the Sequencing Quality Control 2 (SEQC2) project

Accurate indel calling plays an important role in precision medicine. A benchmarking indel set is essential for thoroughly evaluating the indel calling performance of bioinformatics pipelines. A reference sample with a set of known-positive variants was developed in the FDA-led Sequencing Quality Control Phase 2 (SEQC2) project, but the known indels in the known-positive set were limited. This project sought to provide an enriched set of known indels that would be more translationally relevant by focusing on additional cancer related regions. A thorough manual review process completed by 42 reviewers, two advisors, and a judging panel of three researchers significantly enriched the known indel set by an additional 516 indels. The extended benchmarking indel set has a large range of variant allele frequencies (VAFs), with 87% of them having a VAF below 20% in reference Sample A. The reference Sample A and the indel set can be used for comprehensive benchmarking of indel calling across a wider range of VAF values in the lower range. Indel length was also variable, but the majority were under 10 base pairs (bps). Most of the indels were within coding regions, with the remainder in the gene regulatory regions. Although high confidence can be derived from the robust study design and meticulous human review, this extensive indel set has not undergone orthogonal validation. The extended benchmarking indel set, along with the indels in the previously published known-positive set, was the truth set used to benchmark indel calling pipelines in a community challenge hosted on the precisionFDA platform. This benchmarking indel set and reference samples can be utilized for a comprehensive evaluation of indel calling pipelines. Additionally, the insights and solutions obtained during the manual review process can aid in improving the performance of these pipelines.

RxNorm for drug name normalization: a case study of prescription opioids in the FDA adverse events reporting system

Numerous studies have been conducted on the US Food and Drug Administration (FDA) Adverse Events Reporting System (FAERS) database to assess post-marketing reporting rates for drug safety review and risk assessment. However, the drug names in the adverse event (AE) reports from FAERS were heterogeneous due to a lack of uniformity of information submitted mandatorily by pharmaceutical companies and voluntarily by patients, healthcare professionals, and the public. Studies using FAERS and other spontaneous reporting AEs database without drug name normalization may encounter incomplete collection of AE reports from non-standard drug names and the accuracies of the results might be impacted. In this study, we demonstrated applicability of RxNorm, developed by the National Library of Medicine, for drug name normalization in FAERS. Using prescription opioids as a case study, we used RxNorm application program interface (API) to map all FDA-approved prescription opioids described in FAERS AE reports to their equivalent RxNorm Concept Unique Identifiers (RxCUIs) and RxNorm names. The different names of the opioids were then extracted, and their usage frequencies were calculated in collection of more than 14.9 million AE reports for 13 FDA-approved prescription opioid classes, reported over 17 years. The results showed that a significant number of different names were consistently used for opioids in FAERS reports, with 2,086 different names (out of 7,892) used at least three times and 842 different names used at least ten times for each of the 92 RxNorm names of FDA-approved opioids. Our method of using RxNorm API mapping was confirmed to be efficient and accurate and capable of reducing the heterogeneity of prescription opioid names significantly in the AE reports in FAERS; meanwhile, it is expected to have a broad application to different sets of drug names from any database where drug names are diverse and unnormalized. It is expected to be able to automatically standardize and link different representations of the same drugs to build an intact and high-quality database for diverse research, particularly postmarketing data analysis in pharmacovigilance initiatives.

Quartet DNA reference materials and datasets for comprehensively evaluating germline variant calling performance

BACKGROUND

Genomic DNA reference materials are widely recognized as essential for ensuring data quality in omics research. However, relying solely on reference datasets to evaluate the accuracy of variant calling results is incomplete, as they are limited to benchmark regions. Therefore, it is important to develop DNA reference materials that enable the assessment of variant detection performance across the entire genome.

RESULTS

We established a DNA reference material suite from four immortalized cell lines derived from a family of parents and monozygotic twins. Comprehensive reference datasets of 4.2 million small variants and 15,000 structural variants were integrated and certified for evaluating the reliability of germline variant calls inside the benchmark regions. Importantly, the genetic built-in-truth of the Quartet family design enables estimation of the precision of variant calls outside the benchmark regions. Using the Quartet reference materials along with study samples, batch effects are objectively monitored and alleviated by training a machine learning model with the Quartet reference datasets to remove potential artifact calls. Moreover, the matched RNA and protein reference materials and datasets from the Quartet project enables cross-omics validation of variant calls from multiomics data.

CONCLUSIONS

The Quartet DNA reference materials and reference datasets provide a unique resource for objectively assessing the quality of germline variant calls throughout the whole-genome regions and improving the reliability of large-scale genomic profiling.

A generative adversarial network model alternative to animal studies for clinical pathology assessment

Animal studies are unavoidable in evaluating chemical and drug safety. Generative Adversarial Networks (GANs) can generate synthetic animal data by learning from the legacy animal study results, thus may serve as an alternative approach to assess untested chemicals. AnimalGAN, a GAN method to simulate 38 rat clinical pathology measures, was developed with significant robustness even for the drugs that vary significantly from these used during training, both in terms of chemical structure, drug class, and the year of FDA approval. AnimalGAN showed comparable results in hepatotoxicity assessment as using the real animal data and outperformed 12 conventional quantitative structure-activity relationship approaches. Using AnimalGAN, a virtual experiment of 100,000 rats ranked hepatotoxicity of three structurally similar drugs in a similar trend that has been observed in human population. AnimalGAN represented a significant step with artificial intelligence towards the global effort in replacement, reduction, and refinement (3Rs) of animal use.

DICTrank: The largest reference list of 1318 human drugs ranked by risk of drug-induced cardiotoxicity using FDA labeling

Drug-induced cardiotoxicity (DICT) is a leading cause of drug trial failure and discontinuation. Current drug annotations for cardiotoxicity largely focus on individual outcomes or mechanisms. Considering the broad spectrum of adverse cardiac events, we developed Drug-Induced Cardiotoxicity Rank (DICTrank) using FDA labeling and comprehensively classified 1318 human drugs into four categories: Most-DICT-Concern (n = 341), Less-DICT-Concern (n = 528), No-DICT-Concern (n = 343), and Ambiguous-DICT-Concern (n = 106). Notably, DICTrank covers diverse therapeutic categories, of which several were enriched with Most-DICT-Concern drugs, such as antineoplastic agents, sex hormones, anti-inflammatory drugs, beta-blockers, and cardiac therapy. DICTrank currently presents the largest drug list of DICT annotation, and it could contribute to the development of new approach methods, including AI models for early identification of DICT risk during drug development and beyond.

RxBERT: Enhancing drug labeling text mining and analysis with AI language modeling

The US drug labeling document contains essential information on drug efficacy and safety, making it a crucial regulatory resource for Food and Drug Administration (FDA) drug reviewers. Due to its extensive volume and the presence of free-text, conventional text mining analysis have encountered challenges in processing these data. Recent advances in artificial intelligence (AI) for natural language processing (NLP) have provided an unprecedented opportunity to identify key information from drug labeling, thereby enhancing safety reviews and support for regulatory decisions. We developed RxBERT, a Bidirectional Encoder Representations from Transformers (BERT) model pretrained on FDA human prescription drug labeling documents for an enhanced application of drug labeling documents in both research and drug review. RxBERT was derived from BioBERT with further training on human prescription drug labeling documents. RxBERT was demonstrated in several tasks using regulatory datasets, including those involved in the National Institutes of Technology Text Analysis Challenge Dataset (NIST TAC dataset), the FDA Adverse Drug Event Evaluation Dataset (ADE Eval dataset), and the classification of texts from submission packages into labeling sections (US Drug Labeling dataset). For all these tasks, RxBERT reached 86.5 F1-scores in both TAC and ADE Eval classification, respectively, and prediction accuracy of 87% for the US Drug Labeling dataset. Overall, RxBERT was shown to be as competitive or have better performance compared to other NLP approaches such as BERT, BioBERT, etc. In summary, we developed RxBERT, a transformer-based model specific for drug labeling that outperformed the original BERT model. RxBERT has the potential to be used to assist research scientists and FDA reviewers to better process and utilize drug labeling information toward the advancement of drug effectiveness and safety for public health. This proof-of-concept study also demonstrated a potential pathway to customized large language models (LLMs) tailored to the sensitive regulatory documents for internal application.

PLM-ARG: antibiotic resistance gene identification using a pretrained protein language model

MOTIVATION

Antibiotic resistance presents a formidable global challenge to public health and the environment. While considerable endeavors have been dedicated to identify antibiotic resistance genes (ARGs) for assessing the threat of antibiotic resistance, recent extensive investigations using metagenomic and metatranscriptomic approaches have unveiled a noteworthy concern. A significant fraction of proteins defies annotation through conventional sequence similarity-based methods, an issue that extends to ARGs, potentially leading to their under-recognition due to dissimilarities at the sequence level.

RESULTS

Herein, we proposed an Artificial Intelligence-powered ARG identification framework using a pretrained large protein language model, enabling ARG identification and resistance category classification simultaneously. The proposed PLM-ARG was developed based on the most comprehensive ARG and related resistance category information (>28K ARGs and associated 29 resistance categories), yielding Matthew's correlation coefficients (MCCs) of 0.983 ± 0.001 by using a 5-fold cross-validation strategy. Furthermore, the PLM-ARG model was verified using an independent validation set and achieved an MCC of 0.838, outperforming other publicly available ARG prediction tools with an improvement range of 51.8%-107.9%. Moreover, the utility of the proposed PLM-ARG model was demonstrated by annotating resistance in the UniProt database and evaluating the impact of ARGs on the Earth's environmental microbiota.

AVAILABILITY AND IMPLEMENTATION

PLM-ARG is available for academic purposes at https://github.com/Junwu302/PLM-ARG, and a user-friendly webserver (http://www.unimd.org/PLM-ARG) is also provided.

A systematic analysis and data mining of opioid-related adverse events submitted to the FAERS database

The opioid epidemic has become a serious national crisis in the United States. An indepth systematic analysis of opioid-related adverse events (AEs) can clarify the risks presented by opioid exposure, as well as the individual risk profiles of specific opioid drugs and the potential relationships among the opioids. In this study, 92 opioids were identified from the list of all Food and Drug Administration (FDA)-approved drugs, annotated by RxNorm and were classified into 13 opioid groups: buprenorphine, codeine, dihydrocodeine, fentanyl, hydrocodone, hydromorphone, meperidine, methadone, morphine, oxycodone, oxymorphone, tapentadol, and tramadol. A total of 14,970,399 AE reports were retrieved and downloaded from the FDA Adverse Events Reporting System (FAERS) from 2004, Quarter 1 to 2020, Quarter 3. After data processing, Empirical Bayes Geometric Mean (EBGM) was then applied which identified 3317 pairs of potential risk signals within the 13 opioid groups. Based on these potential safety signals, a comparative analysis was pursued to provide a global overview of opioid-related AEs for all 13 groups of FDA-approved prescription opioids. The top 10 most reported AEs for each opioid class were then presented. Both network analysis and hierarchical clustering analysis were conducted to further explore the relationship between opioids. Results from the network analysis revealed a close association among fentanyl, oxycodone, hydrocodone, and hydromorphone, which shared more than 22 AEs. In addition, much less commonly reported AEs were shared among dihydrocodeine, meperidine, oxymorphone, and tapentadol. On the contrary, the hierarchical clustering analysis further categorized the 13 opioid classes into two groups by comparing the full profiles of presence/absence of AEs. The results of network analysis and hierarchical clustering analysis were not only consistent and cross-validated each other but also provided a better and deeper understanding of the associations and relationships between the 13 opioid groups with respect to their adverse effect profiles.

Bidirectional Encoder Representations from Transformers-like large language models in patient safety and pharmacovigilance: A comprehensive assessment of causal inference implications

Causality assessment is vital in patient safety and pharmacovigilance (PSPV) for safety signal detection, adverse reaction management, and regulatory submission. Large language models (LLMs), especially those designed with transformer architecture, are revolutionizing various fields, including PSPV. While attempts to utilize Bidirectional Encoder Representations from Transformers (BERT)-like LLMs for causal inference in PSPV are underway, a detailed evaluation of "fit-for-purpose" BERT-like model selection to enhance causal inference performance within PSPV applications remains absent. This study conducts an in-depth exploration of BERT-like LLMs, including generic pre-trained BERT LLMs, domain-specific pre-trained LLMs, and domain-specific pre-trained LLMs with safety knowledge-specific fine-tuning, for causal inference in PSPV. Our investigation centers around (1) the influence of data complexity and model architecture, (2) the correlation between the BERT size and its impact, and (3) the role of domain-specific training and fine-tuning on three publicly accessible PSPV data sets. The findings suggest that (1) BERT-like LLMs deliver consistent predictive power across varied data complexity levels, (2) the predictive performance and causal inference results do not directly correspond to the BERT-like model size, and (3) domain-specific pre-trained LLMs, with or without safety knowledge-specific fine-tuning, surpass generic pre-trained BERT models in causal inference. The findings are valuable to guide the future application of LLMs in a broad range of application.

The Quartet Data Portal: integration of community-wide resources for multiomics quality control

The Quartet Data Portal facilitates community access to well-characterized reference materials, reference datasets, and related resources established based on a family of four individuals with identical twins from the Quartet Project. Users can request DNA, RNA, protein, and metabolite reference materials, as well as datasets generated across omics, platforms, labs, protocols, and batches. Reproducible analysis tools allow for objective performance assessment of user-submitted data, while interactive visualization tools support rapid exploration of reference datasets. A closed-loop "distribution-collection-evaluation-integration" workflow enables updates and integration of community-contributed multiomics data. Ultimately, this portal helps promote the advancement of reference datasets and multiomics quality control.

Evaluation of QSAR models for predicting mutagenicity: outcome of the Second Ames/QSAR international challenge project

Quantitative structure-activity relationship (QSAR) models are powerful in silico tools for predicting the mutagenicity of unstable compounds, impurities and metabolites that are difficult to examine using the Ames test. Ideally, Ames/QSAR models for regulatory use should demonstrate high sensitivity, low false-negative rate and wide coverage of chemical space. To promote superior model development, the Division of Genetics and Mutagenesis, National Institute of Health Sciences, Japan (DGM/NIHS), conducted the Second Ames/QSAR International Challenge Project (2020-2022) as a successor to the First Project (2014-2017), with 21 teams from 11 countries participating. The DGM/NIHS provided a curated training dataset of approximately 12,000 chemicals and a trial dataset of approximately 1,600 chemicals, and each participating team predicted the Ames mutagenicity of each trial chemical using various Ames/QSAR models. The DGM/NIHS then provided the Ames test results for trial chemicals to assist in model improvement. Although overall model performance on the Second Project was not superior to that on the First, models from the eight teams participating in both projects achieved higher sensitivity than models from teams participating in only the Second Project. Thus, these evaluations have facilitated the development of QSAR models.

DeepAmes: A deep learning-powered Ames test predictive model with potential for regulatory application

The Ames assay is required by the regulatory agencies worldwide to assess the mutagenic potential risk of consumer products. As well as this in vitro assay, in silico approaches have been widely used to predict Ames test results as outlined in the International Council for Harmonization (ICH) guidelines. Building on this in silico approach, here we describe DeepAmes, a high performance and robust model developed with a novel deep learning (DL) approach for potential utility in regulatory science. DeepAmes was developed with a large and consistent Ames dataset (>10,000 compounds) and was compared with other five standard Machine Learning (ML) methods. Using a test set of 1,543 compounds, DeepAmes was the best performer in predicting the outcome of Ames assay. In addition, DeepAmes yielded the best and most stable performance up to when compounds were >30% outside of the applicability domain (AD). Regarding the potential for regulatory application, a revised version of DeepAmes with a much-improved sensitivity of 0.87 from 0.47. In conclusion, DeepAmes provides a DL-powered Ames test predictive model for predicting the results of Ames tests; with its defined AD and clear context of use, DeepAmes has potential for utility in regulatory application.

Quartet RNA reference materials improve the quality of transcriptomic data through ratio-based profiling

Certified RNA reference materials are indispensable for assessing the reliability of RNA sequencing to detect intrinsically small biological differences in clinical settings, such as molecular subtyping of diseases. As part of the Quartet Project for quality control and data integration of multi-omics profiling, we established four RNA reference materials derived from immortalized B-lymphoblastoid cell lines from four members of a monozygotic twin family. Additionally, we constructed ratio-based transcriptome-wide reference datasets between two samples, providing cross-platform and cross-laboratory 'ground truth'. Investigation of the intrinsically subtle biological differences among the Quartet samples enables sensitive assessment of cross-batch integration of transcriptomic measurements at the ratio level. The Quartet RNA reference materials, combined with the ratio-based reference datasets, can serve as unique resources for assessing and improving the quality of transcriptomic data in clinical and biological settings.

Multi-omics data integration using ratio-based quantitative profiling with Quartet reference materials

Characterization and integration of the genome, epigenome, transcriptome, proteome and metabolome of different datasets is difficult owing to a lack of ground truth. Here we develop and characterize suites of publicly available multi-omics reference materials of matched DNA, RNA, protein and metabolites derived from immortalized cell lines from a family quartet of parents and monozygotic twin daughters. These references provide built-in truth defined by relationships among the family members and the information flow from DNA to RNA to protein. We demonstrate how using a ratio-based profiling approach that scales the absolute feature values of a study sample relative to those of a concurrently measured common reference sample produces reproducible and comparable data suitable for integration across batches, labs, platforms and omics types. Our study identifies reference-free 'absolute' feature quantification as the root cause of irreproducibility in multi-omics measurement and data integration and establishes the advantages of ratio-based multi-omics profiling with common reference materials.

Correcting batch effects in large-scale multiomics studies using a reference-material-based ratio method

BACKGROUND

Batch effects are notoriously common technical variations in multiomics data and may result in misleading outcomes if uncorrected or over-corrected. A plethora of batch-effect correction algorithms are proposed to facilitate data integration. However, their respective advantages and limitations are not adequately assessed in terms of omics types, the performance metrics, and the application scenarios.

RESULTS

As part of the Quartet Project for quality control and data integration of multiomics profiling, we comprehensively assess the performance of seven batch effect correction algorithms based on different performance metrics of clinical relevance, i.e., the accuracy of identifying differentially expressed features, the robustness of predictive models, and the ability of accurately clustering cross-batch samples into their own donors. The ratio-based method, i.e., by scaling absolute feature values of study samples relative to those of concurrently profiled reference material(s), is found to be much more effective and broadly applicable than others, especially when batch effects are completely confounded with biological factors of study interests. We further provide practical guidelines for implementing the ratio based approach in increasingly large-scale multiomics studies.

CONCLUSIONS

Multiomics measurements are prone to batch effects, which can be effectively corrected using ratio-based scaling of the multiomics data. Our study lays the foundation for eliminating batch effects at a ratio scale.

Classifying Free Texts Into Predefined Sections Using AI in Regulatory Documents: A Case Study with Drug Labeling Documents

The US Food and Drug Administration (FDA) regulatory process often involves several reviewers who focus on sets of information related to their respective areas of review. Accordingly, manufacturers that provide submission packages to regulatory agencies are instructed to organize the contents using a structure that enables the information to be easily allocated, retrieved, and reviewed. However, this practice is not always followed correctly; as such, some documents are not well structured, with similar information spreading across different sections, hindering the efficient access and review of all of the relevant data as a whole. To improve this common situation, we evaluated an artificial intelligence (AI)-based natural language processing (NLP) methodology, called Bidirectional Encoder Representations from Transformers (BERT), to automatically classify free-text information into standardized sections, supporting a holistic review of drug safety and efficacy. Specifically, FDA labeling documents were used in this study as a proof of concept, where the labeling section structure defined by the Physician Label Rule (PLR) was used to classify labels in the development of the model. The model was subsequently evaluated on texts from both well-structured labeling documents (i.e., PLR-based labeling) and less- or differently structured documents (i.e., non-PLR and Summary of Product Characteristic [SmPC] labeling.) In the training process, the model yielded 96% and 88% accuracy for binary and multiclass tasks, respectively. The testing accuracies observed for the PLR, non-PLR, and SmPC testing data sets for the binary model were 95%, 88%, and 88%, and for the multiclass model were 82%, 73%, and 68%, respectively. Our study demonstrated that automatically classifying free texts into standardized sections with AI language models could be an advanced regulatory science approach for supporting the review process by effectively processing unformatted documents.

A Weakly Supervised Deep Learning Framework for Whole Slide Classification to Facilitate Digital Pathology in Animal Study

The pathology of animal studies is crucial for toxicity evaluations and regulatory assessments, but the manual examination of slides by pathologists remains time-consuming and requires extensive training. One inherent challenge in this process is the interobserver variability, which can compromise the consistency and accuracy of a study. Artificial intelligence (AI) has demonstrated its ability to automate similar examinations in clinical applications with enhanced efficiency, consistency, and accuracy. However, training AI models typically relies on costly pixel-level annotation of injured regions and is often not available for animal pathology. To address this, we developed the PathologAI system, a "weakly" supervised approach for WSI classification in rat images without explicit lesion annotation at the pixel level. Using rat liver imaging data from the Open TG-GATEs system, PathologAI was applied to predict necrosis of n = 816 WSIs (377 controls). TG-GATEs studied 170 compounds at three dose levels (low, middle, and high) for four time points (3, 7, 14, and 28 days). PathologAI first preprocessed WSIs at the tile level to generate a high-level representation with a Generative Adversarial Network architecture. The prediction of liver necrosis relied on an ensemble model of 5 CNN classifiers trained on 335 WSIs. The ensemble model achieved notable classification accuracy on the holdout test set: 87% among 87 control slides free of findings, 83% among 120 controls with spontaneous necrosis, 67% among 147 treated animals with spontaneous minimal or slight necrosis, and 59% among 127 treated animals with minimal or slight necrosis caused by the treatment. Importantly, PathologAI was able to discriminate WSIs with spontaneous necrosis from those with treatment related necrosis and discriminated mild lesion level findings (slight vs minimal) and between treatment dose levels. PathologAI could provide an inexpensive and rapid screening tool to assist the digital pathology analysis in preclinical applications and general toxicological studies.

TransOrGAN: An Artificial Intelligence Mapping of Rat Transcriptomic Profiles between Organs, Ages, and Sexes

Animal studies are required for the evaluation of candidate drugs to ensure patient and volunteer safety. Toxicogenomics is often applied in these studies to gain understanding of the underlying mechanisms of toxicity, which is usually focused on critical organs such as the liver or kidney in young male rats. There is a strong ethical reason to reduce, refine and replace animal use (the 3Rs), where the mapping of data between organs, sexes and ages could reduce the cost and time of drug development. Herein, we proposed a generative adversarial network (GAN)-based framework entitled TransOrGAN that allowed the molecular mapping of gene expression profiles in different rodent organ systems and across sex and age groups. We carried out a proof-of-concept study based on rat RNA-seq data from 288 samples in 9 different organs of both sexes and 4 developmental stages. First, we demonstrated that TransOrGAN could infer transcriptomic profiles between any 2 of the 9 organs studied, yielding an average cosine similarity of 0.984 between synthetic transcriptomic profiles and their corresponding real profiles. Second, we found that TransOrGAN could infer transcriptomic profiles observed in females from males, with an average cosine similarity of 0.984. Third, we found that TransOrGAN could infer transcriptomic profiles in juvenile, adult, and aged animals from adolescent animals with an average cosine similarity of 0.981, 0.983, and 0.989, respectively. Altogether, TransOrGAN is an innovative approach to infer transcriptomic profiles between ages, sexes, and organ systems, offering the opportunity to reduce animal usage and to provide an integrated assessment of toxicity in the whole organism irrespective of sex or age.

Artificial intelligence and real-world data for drug and food safety - A regulatory science perspective

In 2013, the Global Coalition for Regulatory Science Research (GCRSR) was established with members from over ten countries (www.gcrsr.net). One of the main objectives of GCRSR is to facilitate communication among global regulators on the rise of new technologies with regulatory applications through the annual conference Global Summit on Regulatory Science (GSRS). The 11th annual GSRS conference (GSRS21) focused on "Regulatory Sciences for Food/Drug Safety with Real-World Data (RWD) and Artificial Intelligence (AI)." The conference discussed current advancements in both AI and RWD approaches with a specific emphasis on how they impact regulatory sciences and how regulatory agencies across the globe are pursuing the adaptation and oversight of these technologies. There were presentations from Brazil, Canada, India, Italy, Japan, Germany, Switzerland, Singapore, the United Kingdom, and the United States. These presentations highlighted how various agencies are moving forward with these technologies by either improving the agencies' operation and/or preparing regulatory mechanisms to approve the products containing these innovations. To increase the content and discussion, the GSRS21 hosted two debate sessions on the question of "Is Regulatory Science Ready for AI?" and a workshop to showcase the analytical data tools that global regulatory agencies have been using and/or plan to apply to regulatory science. Several key topics were highlighted and discussed during the conference, such as the capabilities of AI and RWD to assist regulatory science policies for drug and food safety, the readiness of AI and data science to provide solutions for regulatory science. Discussions highlighted the need for a constant effort to evaluate emerging technologies for fit-for-purpose regulatory applications. The annual GSRS conferences offer a unique platform to facilitate discussion and collaboration across regulatory agencies, modernizing regulatory approaches, and harmonizing efforts.

Robotic versus laparoscopic left colectomy with complete mesocolic excision for left-sided colon cancer: a multicentre study with propensity score matching analysis

BACKGROUND

Robotic surgery for right-sided colon and rectal cancer has rapidly increased; however, there is limited evidence in the literature of advantages of robotic left colectomy (RLC) for left-sided colon cancer. The purpose of this study was to compare the outcomes of RLC versus laparoscopic left colectomy (LLC) with complete mesocolic excision (CME) for left-sided colon cancer.

METHODS

Patients who had RLC or LLC with CME for left-sided colon cancer at 5 hospitals in China between January 2014 and April 2022 were included. A one-to-one propensity score matched analysis was performed to decrease confounding. The primary outcome was postoperative complications occurring within 30 days of surgery. Secondary outcomes were disease-free survival, overall survival and the number of harvested lymph nodes.

RESULTS

A total of 292 patients (187 males; median age 61.0 [20.0-85.0] years) were eligible for this study, and propensity score matching yielded 102 patients in each group. The clinical-pathological characteristics were well-matched between groups. The two groups did not differ in estimated blood loss, conversion to open rate, time to first flatus, reoperation rate, or postoperative length of hospital stay (p > 0.05). RLC was associated with a longer operation time (192.9 ± 53.2 vs. 168.9 ± 52.8 minutes, p=0.001). The incidence of postoperative complications did not differ between the RLC and LLC groups (18.6% vs. 17.6%, p = 0.856). The total number of lymph nodes harvested in the RLC group was higher than that in the LLC group (15.7 ± 8.3 vs. 12.1 ± 5.9, p< 0.001). There were no significant differences in 3-year and 5-year overall survival or 3-year and 5-year disease-free survival.

CONCLUSIONS

Compared to laparoscopic surgery, RLC with CME for left-sided colon cancer was found to be associated with higher numbers of lymph nodes harvested and similar postoperative complications and long-term survival outcomes.

DeepCausality: A general AI-powered causal inference framework for free text: A case study of LiverTox

Causality plays an essential role in multiple scientific disciplines, including the social, behavioral, and biological sciences and portions of statistics and artificial intelligence. Manual-based causality assessment from a large number of free text-based documents is very time-consuming, labor-intensive, and sometimes even impractical. Herein, we proposed a general causal inference framework named DeepCausality to empirically estimate the causal factors for suspected endpoints embedded in the free text. The proposed DeepCausality seamlessly incorporates AI-powered language models, named entity recognition and Judea Pearl's Do-calculus, into a general framework for causal inference to fulfill different domain-specific applications. We exemplified the utility of the proposed DeepCausality framework by employing the LiverTox database to estimate idiosyncratic drug-induced liver injury (DILI)-related causal terms and generate a knowledge-based causal tree for idiosyncratic DILI patient stratification. Consequently, the DeepCausality yielded a prediction performance with an accuracy of 0.92 and an F-score of 0.84 for the DILI prediction. Notably, 90% of causal terms enriched by the DeepCausality were consistent with the clinical causal terms defined by the American College of Gastroenterology (ACG) clinical guideline for evaluating suspected idiosyncratic DILI (iDILI). Furthermore, we observed a high concordance of 0.91 between the iDILI severity scores generated by DeepCausality and domain experts. Altogether, the proposed DeepCausality framework could be a promising solution for causality assessment from free text and is publicly available through https://github.com/XingqiaoWang/https-github.com-XingqiaoWang-DeepCausality-LiverTox.

Development of benchmark datasets for text mining and sentiment analysis to accelerate regulatory literature review

In the field of regulatory science, reviewing literature is an essential and important step, which most of the time is conducted by manually reading hundreds of articles. Although this process is highly time-consuming and labor-intensive, most output of this process is not well transformed into machine-readable format. The limited availability of data has largely constrained the artificial intelligence (AI) system development to facilitate this literature reviewing in the regulatory process. In the past decade, AI has revolutionized the area of text mining as many deep learning approaches have been developed to search, annotate, and classify relevant documents. After the great advancement of AI algorithms, a lack of high-quality data instead of the algorithms has recently become the bottleneck of AI system development. Herein, we constructed two large benchmark datasets, Chlorine Efficacy dataset (CHE) and Chlorine Safety dataset (CHS), under a regulatory scenario that sought to assess the antiseptic efficacy and toxicity of chlorine. For each dataset, ∼10,000 scientific articles were initially collected, manually reviewed, and their relevance to the review task were labeled. To ensure high data quality, each paper was labeled by a consensus among multiple experienced reviewers. The overall relevance rate was 27.21% (2,663 of 9,788) for CHE and 7.50% (761 of 10,153) for CHS, respectively. Furthermore, the relevant articles were categorized into five subgroups based on the focus of their content. Next, we developed an attention-based classification language model using these two datasets. The proposed classification model yielded 0.857 and 0.908 of Area Under the Curve (AUC) for CHE and CHS dataset, respectively. This performance was significantly better than permutation test (p < 10E-9), demonstrating that the labeling processes were valid. To conclude, our datasets can be used as benchmark to develop AI systems, which can further facilitate the literature review process in regulatory science.

Adaptability of AI for safety evaluation in regulatory science: A case study of drug-induced liver injury

Artificial intelligence (AI) has played a crucial role in advancing biomedical sciences but has yet to have the impact it merits in regulatory science. As the field advances, in silico and in vitro approaches have been evaluated as alternatives to animal studies, in a drive to identify and mitigate safety concerns earlier in the drug development process. Although many AI tools are available, their acceptance in regulatory decision-making for drug efficacy and safety evaluation is still a challenge. It is a common perception that an AI model improves with more data, but does reality reflect this perception in drug safety assessments? Importantly, a model aiming at regulatory application needs to take a broad range of model characteristics into consideration. Among them is adaptability, defined as the adaptive behavior of a model as it is retrained on unseen data. This is an important model characteristic which should be considered in regulatory applications. In this study, we set up a comprehensive study to assess adaptability in AI by mimicking the real-world scenario of the annual addition of new drugs to the market, using a model we previously developed known as DeepDILI for predicting drug-induced liver injury (DILI) with a novel Deep Learning method. We found that the target test set plays a major role in assessing the adaptive behavior of our model. Our findings also indicated that adding more drugs to the training set does not significantly affect the predictive performance of our adaptive model. We concluded that the proposed adaptability assessment framework has utility in the evaluation of the performance of a model over time.

NeuroCORD: A Language Model to Facilitate COVID-19-Associated Neurological Disorder Studies

COVID-19 can lead to multiple severe outcomes including neurological and psychological impacts. However, it is challenging to manually scan hundreds of thousands of COVID-19 articles on a regular basis. To update our knowledge, provide sound science to the public, and communicate effectively, it is critical to have an efficient means of following the most current published data. In this study, we developed a language model to search abstracts using the most advanced artificial intelligence (AI) to accurately retrieve articles on COVID-19-associated neurological disorders. We applied this NeuroCORD model to the largest benchmark dataset of COVID-19, CORD-19. We found that the model developed on the training set yielded 94% prediction accuracy on the test set. This result was subsequently verified by two experts in the field. In addition, when applied to 96,000 non-labeled articles that were published after 2020, the NeuroCORD model accurately identified approximately 3% of them to be relevant for the study of COVID-19-associated neurological disorders, while only 0.5% were retrieved using conventional keyword searching. In conclusion, NeuroCORD provides an opportunity to profile neurological disorders resulting from COVID-19 in a rapid and efficient fashion, and its general framework could be used to study other COVID-19-related emerging health issues.

Accurate species identification of food-contaminating beetles with quality-improved elytral images and deep learning

Food samples are routinely screened for food-contaminating beetles (i.e., pantry beetles) due to their adverse impact on the economy, environment, public health and safety. If found, their remains are subsequently analyzed to identify the species responsible for the contamination; each species poses different levels of risk, requiring different regulatory and management steps. At present, this identification is done through manual microscopic examination since each species of beetle has a unique pattern on its elytra (hardened forewing). Our study sought to automate the pattern recognition process through machine learning. Such automation will enable more efficient identification of pantry beetle species and could potentially be scaled up and implemented across various analysis centers in a consistent manner. In our earlier studies, we demonstrated that automated species identification of pantry beetles is feasible through elytral pattern recognition. Due to poor image quality, however, we failed to achieve prediction accuracies of more than 80%. Subsequently, we modified the traditional imaging technique, allowing us to acquire high-quality elytral images. In this study, we explored whether high-quality elytral images can truly achieve near-perfect prediction accuracies for 27 different species of pantry beetles. To test this hypothesis, we developed a convolutional neural network (CNN) model and compared performance between two different image sets for various pantry beetles. Our study indicates improved image quality indeed leads to better prediction accuracy; however, it was not the only requirement for achieving good accuracy. Also required are many high-quality images, especially for species with a high number of variations in their elytral patterns. The current study provided a direction toward achieving our ultimate goal of automated species identification through elytral pattern recognition.

Best practice and reproducible science are required to advance artificial intelligence in real-world applications

Drug-induced liver injury (DILI) is one of the most significant concerns in medical practice but yet it still cannot be fully recapitulated with existing in vivo, in vitro and in silico approaches. To address this challenge, Chen et al. [ 1] developed a deep learning-based DILI prediction model based on chemical structure information alone. The reported model yielded an outstanding prediction performance (i.e. 0.958, 0.976, 0.935, 0.947, 0.926 and 0.913 for AUC, accuracy, recall, precision, F1-score and specificity, respectively, on a test set), far outperforming all publicly available and similar in silico DILI models. This extraordinary model performance is counter-intuitive to what we know about the underlying biology of DILI and the principles and hypothesis behind this type of in silico approach. In this Letter to the Editor, we raise awareness of several issues concerning data curation, model validation and comparison practices, and data and model reproducibility.

Deep oncopanel sequencing reveals within block position-dependent quality degradation in FFPE processed samples

Background

Clinical laboratories routinely use formalin-fixed paraffin-embedded (FFPE) tissue or cell block cytology samples in oncology panel sequencing to identify mutations that can predict patient response to targeted therapy. To understand the technical error due to FFPE processing, a robustly characterized diploid cell line was used to create FFPE samples with four different pre-tissue processing formalin fixation times. A total of 96 FFPE sections were then distributed to different laboratories for targeted sequencing analysis by four oncopanels, and variants resulting from technical error were identified.

Results

Tissue sections that fail more frequently show low cellularity, lower than recommended library preparation DNA input, or target sequencing depth. Importantly, sections from block surfaces are more likely to show FFPE-specific errors, akin to “edge effects” seen in histology, while the inner samples display no quality degradation related to fixation time.

Conclusions

To assure reliable results, we recommend avoiding the block surface portion and restricting mutation detection to genomic regions of high confidence.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13059-022-02709-8.

Angular dependence of spin-flop transition in triangular lattice antiferromagnet Cu2(OH)3Br

We report angular dependence of spin-flop transition in triangular lattice antiferromagnet Cu₂(OH)₃Br by angle-dependent magnetization and ESR measurements. The results show that the antiferromagnetic easy magnetization axis is the diagonal direction (θ= 45°) of theac^*plane, i.e., the orientation of Cu1 spins based on the magnetic structure (2020Phys. Rev. Lett.125037204), whereas the spin-flop axis is thebaxis. A phenomenological model is proposed to describe the angle-dependent spin-flop transitions. Based on this model, Cu1 spins are sensitive to external magnetic field, while Cu2 spins are robust against to the field, showing partial decoupling. The model is expected to be used in other uniaxial antiferromagnets with a more general easy axis and complex spin-flop transitions.

Ultra-deep sequencing data from a liquid biopsy proficiency study demonstrating analytic validity

Recently we reported the accuracy and reproducibility of circulating tumor DNA (ctDNA) assays using a unique set of reference materials, associated analytical framework, and suggested best practices. With the rapid adoption of ctDNA sequencing in precision oncology, it is critical to understand the analytical validity and technical limitations of this cutting-edge and medical-practice-changing technology. The SEQC2 Oncopanel Sequencing Working Group has developed a multi-site, cross-platform study design for evaluating the analytical performance of five industry-leading ctDNA assays. The study used tailor-made reference samples at various levels of input material to assess ctDNA sequencing across 12 participating clinical and research facilities. The generated dataset encompasses multiple key variables, including a broad range of mutation frequencies, sequencing coverage depth, DNA input quantity, etc. It is the most comprehensive public-facing dataset of its kind and provides valuable insights into ultra-deep ctDNA sequencing technology. Eventually the clinical utility of ctDNA assays is required and our proficiency study and corresponding dataset are needed steps towards this goal.

Measurement(s)	Somatic Mutation • spike-in quality control role
Technology Type(s)	Tumor Panel Sequencing
Factor Type(s)	Tumor Panel • DNA Library Input Quantity • Variant Allele Frequency
Sample Characteristic - Organism	Homo sapiens

Towards accurate and reliable resolution of structural variants for clinical diagnosis

Structural variants (SVs) are a major source of human genetic diversity and have been associated with different diseases and phenotypes. The detection of SVs is difficult, and a diverse range of detection methods and data analysis protocols has been developed. This difficulty and diversity make the detection of SVs for clinical applications challenging and requires a framework to ensure accuracy and reproducibility. Here, we discuss current developments in the diagnosis of SVs and propose a roadmap for the accurate and reproducible detection of SVs that includes case studies provided from the FDA-led SEquencing Quality Control Phase II (SEQC-II) and other consortium efforts.

Integrative approaches for studying the role of noncoding RNAs in influencing drug efficacy and toxicity

INTRODUCTION

Drug efficacy and toxicity are important factors for evaluation in drug development. Drug metabolizing enzymes and transporters (DMETs) play an essential role in drug efficacy and toxicity. Noncoding RNAs (ncRNAs) have been implicated to influence inter-individual variations in drug efficacy and safety by regulating DMETs. An efficient strategy is urgently needed to identify and functionally characterize ncRNAs that mediate drug efficacy and toxicity through regulating DMETs.

AREAS COVERED

We outline an integrative strategy to identify ncRNAs that modulate DMETs. We include reliable tools and databases for computational prediction of ncRNA targets with regard to their advantages and limitations. Various biochemical, molecular, and cellular assays are discussed for in vitro experimental verification of the regulatory function of ncRNAs. In vivo approaches for association of ncRNAs with drug treatment and toxicity are also reviewed.

EXPERT OPINION

A streamlined integration of computational prediction and wet-lab validation is important to elucidate mechanisms of ncRNAs in the regulation of DMETs related to drug efficacy and safety. Bioinformatic analyses using open-access tools and databases serve as a powerful booster for ncRNA Research in toxicology. Further refinement of computational algorithms and experimental technologies is needed to improve accuracy and efficiency in ncRNA target identification and characterization.

Combined transabdominal-transanal surgical approach for iatrogenic rectovaginal fistula: two case reports

Rectovaginal fistula (RVF) is a type of anastomotic leakage that may occur after low anterior resection for rectal cancer. The repair of RVF can be challenging because of the scar tissue stenosis and incomplete obstruction. Two patients presented in our department with vaginal faecal discharge almost 7 months after the radical resection of rectal cancer. On vaginal examination, titanium nails related to the rectal surgery were found in the vaginal wall. The patients were diagnosed with RVF. Considering that RVF positions in the patients were high and might adhere to the pelvic tissue, a combined transabdominal-transanal resection and vaginal repair surgery was performed. About 3 months after surgery, both patients underwent colonic closure surgery, with consequent good recovery. A combined transabdominal-transanal approach may provide distinct advantages in surgical repair of difficult cases of RVF.

Using synthetic chromosome controls to evaluate the sequencing of difficult regions within the human genome

BACKGROUND

Next-generation sequencing (NGS) can identify mutations in the human genome that cause disease and has been widely adopted in clinical diagnosis. However, the human genome contains many polymorphic, low-complexity, and repetitive regions that are difficult to sequence and analyze. Despite their difficulty, these regions include many clinically important sequences that can inform the treatment of human diseases and improve the diagnostic yield of NGS.

RESULTS

To evaluate the accuracy by which these difficult regions are analyzed with NGS, we built an in silico decoy chromosome, along with corresponding synthetic DNA reference controls, that encode difficult and clinically important human genome regions, including repeats, microsatellites, HLA genes, and immune receptors. These controls provide a known ground-truth reference against which to measure the performance of diverse sequencing technologies, reagents, and bioinformatic tools. Using this approach, we provide a comprehensive evaluation of short- and long-read sequencing instruments, library preparation methods, and software tools and identify the errors and systematic bias that confound our resolution of these remaining difficult regions.

CONCLUSIONS

This study provides an analytical validation of diagnosis using NGS in difficult regions of the human genome and highlights the challenges that remain to resolve these difficult regions.

Assessing reproducibility of inherited variants detected with short-read whole genome sequencing

BACKGROUND

Reproducible detection of inherited variants with whole genome sequencing (WGS) is vital for the implementation of precision medicine and is a complicated process in which each step affects variant call quality. Systematically assessing reproducibility of inherited variants with WGS and impact of each step in the process is needed for understanding and improving quality of inherited variants from WGS.

RESULTS

To dissect the impact of factors involved in detection of inherited variants with WGS, we sequence triplicates of eight DNA samples representing two populations on three short-read sequencing platforms using three library kits in six labs and call variants with 56 combinations of aligners and callers. We find that bioinformatics pipelines (callers and aligners) have a larger impact on variant reproducibility than WGS platform or library preparation. Single-nucleotide variants (SNVs), particularly outside difficult-to-map regions, are more reproducible than small insertions and deletions (indels), which are least reproducible when > 5 bp. Increasing sequencing coverage improves indel reproducibility but has limited impact on SNVs above 30×.

CONCLUSIONS

Our findings highlight sources of variability in variant detection and the need for improvement of bioinformatics pipelines in the era of precision medicine with WGS.

Emerging technologies and their impact on regulatory science

There is an evolution and increasing need for the utilization of emerging cellular, molecular and in silico technologies and novel approaches for safety assessment of food, drugs, and personal care products. Convergence of these emerging technologies is also enabling rapid advances and approaches that may impact regulatory decisions and approvals. Although the development of emerging technologies may allow rapid advances in regulatory decision making, there is concern that these new technologies have not been thoroughly evaluated to determine if they are ready for regulatory application, singularly or in combinations. The magnitude of these combined technical advances may outpace the ability to assess fit for purpose and to allow routine application of these new methods for regulatory purposes. There is a need to develop strategies to evaluate the new technologies to determine which ones are ready for regulatory use. The opportunity to apply these potentially faster, more accurate, and cost-effective approaches remains an important goal to facilitate their incorporation into regulatory use. However, without a clear strategy to evaluate emerging technologies rapidly and appropriately, the value of these efforts may go unrecognized or may take longer. It is important for the regulatory science field to keep up with the research in these technically advanced areas and to understand the science behind these new approaches. The regulatory field must understand the critical quality attributes of these novel approaches and learn from each other's experience so that workforces can be trained to prepare for emerging global regulatory challenges. Moreover, it is essential that the regulatory community must work with the technology developers to harness collective capabilities towards developing a strategy for evaluation of these new and novel assessment tools.

Machine Learning Models for Predicting Liver Toxicity

Liver toxicity is a major adverse drug reaction that accounts for drug failure in clinical trials and withdrawal from the market. Therefore, predicting potential liver toxicity at an early stage in drug discovery is crucial to reduce costs and the potential for drug failure. However, current in vivo animal toxicity testing is very expensive and time consuming. As an alternative approach, various machine learning models have been developed to predict potential liver toxicity in humans. This chapter reviews current advances in the development and application of machine learning models for prediction of potential liver toxicity in humans and discusses possible improvements to liver toxicity prediction.

Tox-GAN: An AI Approach Alternative to Animal Studies-a Case Study with Toxicogenomics

Animal studies are a critical component in biomedical research, pharmaceutical product development, and regulatory submissions. There is a worldwide effort in toxicology towards "reducing, refining and replacing" (3Rs) animal use. Here, we proposed a deep generative adversarial network (GAN)-based framework capable of deriving new animal results from existing animal studies without additional experiments. To prove the concept, we employed this Tox-GAN framework to generate both gene activities and expression profiles for multiple doses and treatment durations in toxicogenomics (TGx). Using the pre-existing rat liver TGx data from the Open TG-GATEs, we generated Tox-GAN transcriptomic profiles with high similarity (0.997 ± 0.002 in intensity and 0.740 ± 0.082 in fold change) to the corresponding real gene expression profiles. Consequently, Tox-GAN showed an outstanding performance in two critical TGx applications, gaining a molecular understanding of underlying toxicological mechanisms and gene expression-based biomarker development. For the former, over 87% agreement in Gene Ontology was found between Tox-GAN results and real gene expression data. For the latter, the concordance of biomarkers between real and generated data was high in both predictive performance and biomarker genes. We also demonstrated that the Tox-GAN models constructed with TG-GATEs data were capable of generating transcriptomic profiles reported in DrugMatrix. Finally, we demonstrated potential utility for Tox-GAN in aiding chemical-based read-across. To the best of our knowledge, the proposed Tox-GAN model is novel in its ability to generate in vivo transcriptomic profiles at different treatment conditions from chemical structures. Overall, Tox-GAN holds great promise for generating high-quality toxicogenomic profiles without animal experimentation.

BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk

Background & Aims: The United States Food and Drug Administration (FDA) regulates a broad range of consumer products, which account for about 25% of the United States market. The FDA regulatory activities often involve producing and reading of a large number of documents, which is time consuming and labor intensive. To support regulatory science at FDA, we evaluated artificial intelligence (AI)-based natural language processing (NLP) of regulatory documents for text classification and compared deep learning-based models with a conventional keywords-based model.

Methods: FDA drug labeling documents were used as a representative regulatory data source to classify drug-induced liver injury (DILI) risk by employing the state-of-the-art language model BERT. The resulting NLP-DILI classification model was statistically validated with both internal and external validation procedures and applied to the labeling data from the European Medicines Agency (EMA) for cross-agency application.

Results: The NLP-DILI model developed using FDA labeling documents and evaluated by cross-validations in this study showed remarkable performance in DILI classification with a recall of 1 and a precision of 0.78. When cross-agency data were used to validate the model, the performance remained comparable, demonstrating that the model was portable across agencies. Results also suggested that the model was able to capture the semantic meanings of sentences in drug labeling.

Conclusion: Deep learning-based NLP models performed well in DILI classification of drug labeling documents and learned the meanings of complex text in drug labeling. This proof-of-concept work demonstrated that using AI technologies to assist regulatory activities is a promising approach to modernize and advance regulatory science.