Big Data Toolsets to Pharmacometrics: Application of Machine Learning for Time-to-Event Analysis

A Basic Machine Learning Primer for Surgical Research in Congenital Heart Disease

Artificial intelligence and machine learning are rapidly transforming medicine, healthcare, and surgery. Machine learning is a valuable tool for surgeons and researchers in pediatric cardiovascular and thoracic surgery, with innovative applications constantly evolving and expanding. Utilizing machine learning in addition to traditional statistical methods can gain insights into the data and develop more powerful prediction models for improving surgical management and patient outcomes. We provide an accessible introduction to machine learning for surgeons to become familiar with its key essential concepts and architecture, along with a five-step strategy for performing machine learning analyses. With careful study planning using high-quality data, active collaboration between surgeons, researchers, statisticians, and data scientists, and real-world implementation of machine learning algorithms in the clinical setting, machine learning can be a strategic tool for gaining insights into the data in order to improve surgical decision-making, patient risk management, and surgical outcomes.

Gender impact on pancreatic neuroendocrine neoplasm (PanNEN) prognosis according to survival nomograms

PURPOSE

Personalizing care and outcome evaluation are important aims in the field of NEN and nomograms may represent useful tools for clinicians. Of note, gender difference is being progressively more considered in NEN care, as it may also impact on survival. This systematic review aims to describe and analyze the available nomograms on pancreatic NENs (PanNENs) to identify if gender differences are evaluated and if they could impact on patients' management and prognosis.

METHODS

We performed an electronic-based search using PubMed updated until June 2024, summarizing the available evidence of gender impact on PanNEN survival outcomes as emerges from published nomograms.

RESULTS

34 articles were identified regarding prognostic nomograms in PanNEN fields. The most included variables were age, tumor grade, tumor stage, while only 5 papers (14.7%) included sex as one of the key model variables with a significant impact on patients' prognosis. These 5 studies analyzed a total of 18,920 PanNENs. 3 studies found a significant impact of sex on overall survival (OS), whereas the remaining 2 studies showed no significant impact of sex on OS.

CONCLUSIONS

Gender difference is being progressively more considered in PanNEN diagnosis, care and survival. Nomograms represent a potentially useful tool in patients' management and in outcomes prediction in the field of PanNENs. A key role of sex in the prognosis of PanNENs has been found in few models, while definitive conclusions couldn't be drawn. Future studies are needed to finally establish gender impact on PanNEN prognosis.

Integrating Quantitative Methods & Modeling and Analytical Techniques in Reverse Engineering; A Cutting-Edge Strategy in Complex Generic Development

Generic drugs are crucial for healthcare, offering affordable alternatives to brand-name drugs. Complex generics, with intricate ingredients, are gaining increasing importance in managing chronic conditions. However, prior to the regulatory market approval, they must demonstrate similarity in active ingredients, formulations, strength, and administration routes to ensure bioequivalence. The primary constraint lies in demonstrating bioequivalence with the innovator drug using traditional methods includes a lack of advanced technologies, and standardized protocols for analysing complex products. Given the multifaceted nature of these products, a single methodology may not suffice to establish in vitro/in vivo bioequivalence. Recognizing this, the USFDA conducts several workshops aiming advancement of complex generic drug product development. Notably, these efforts highlight the need to use Quantitative Methods and Modeling (QMM) approaches to support generic product development. QMM is a scientific approach used to analyze data and simulate drug development processes, ensuring safe, effective, and similar formulations of generic drugs using mathematical, statistical, and computational tools. QMM facilitates the design of formulations and processes, establishes a framework for in vivo BE studies, and suggests alternative ways to demonstrate BE. Appropriate utilization of the QMM approach can reduce the need for unwanted in vivo studies and bolster in vitro approaches for generic product development. Furthermore, use of orthogonal analytical techniques to characterize and decode innovator drugs can provide valuable insights into product attributes. Integrating this data into QMM enables the assessment of critical material attributes, or critical process parameters, thus demonstrating sameness. The combined application of QMM and analytical techniques not only supports regulatory decisions but also enhances the success rate of complex generic drug products.

Upfront surgery for intrahepatic cholangiocarcinoma: Prediction of futility using artificial intelligence

OBJECTIVE

We sought to identify patients at risk of "futile" surgery for intrahepatic cholangiocarcinoma using an artificial intelligence (AI)-based model based on preoperative variables.

METHODS

Intrahepatic cholangiocarcinoma patients who underwent resection between 1990 and 2020 were identified from a multi-institutional database. Futility was defined either as mortality or recurrence within 12 months of surgery. Various machine learning and deep learning techniques were used to develop prediction models for futile surgery.

RESULTS

Overall, 827 intrahepatic cholangiocarcinoma patients were included. Among 378 patients (45.7%) who had futile surgery, 297 patients (78.6%) developed intrahepatic cholangiocarcinoma recurrence and 81 patients (21.4%) died within 12 months of surgical resection. An ensemble model consisting of multilayer perceptron and gradient boosting classifiers that used 10 preoperative factors demonstrated the highest accuracy, with areas under receiver operating characteristic curves of 0.830 (95% confidence interval 0.798-0.861) and 0.781 (95% confidence interval 0.707-0.853) in the training and testing cohorts, respectively. The model displayed sensitivity and specificity of 64.5% and 80.0%, respectively, with positive and negative predictive values of 73.1% and 72.7%, respectively. Radiologic tumor burden score, serum carbohydrate antigen 19-9, and direct bilirubin levels were the factors most strongly predictive of futile surgery. The artificial intelligence-based model was made available online for ease of use and clinical applicability (https://altaf-pawlik-icc-futilityofsurgery-calculator.streamlit.app/).

CONCLUSION

The artificial intelligence ensemble model demonstrated high accuracy to identify patients preoperatively at high risk of undergoing futile surgery for intrahepatic cholangiocarcinoma. Artificial intelligence-based prediction models can provide clinicians with reliable preoperative guidance and aid in avoiding futile surgical procedures that are unlikely to provide patients long-term benefits.

Clinical Pharmacology of Cytokine Release Syndrome with T-Cell-Engaging Bispecific Antibodies: Current Insights and Drug Development Strategies

Cytokine release syndrome (CRS) is a common acute toxicity in T-cell therapies, including T-cell-engaging bispecific antibodies (T-BiSp). Effective CRS management and prevention are crucial in T-BiSp development. Required hospitalization for seven of the nine approved T-BiSp and the need for clinical intervention in severe cases highlight the importance of mitigation strategies to reduce health care burden and improve patient outcomes. In this review, we discuss the emerging evidence on CRS mitigation, management, and prediction. We cover different strategies for dose optimization, current and emerging (pre) treatment strategies, quantitative pharmacology tools used during drug development, and biomarkers and predictive factors. Insights are gleaned on step-up dosing and formulation effects on CRS and CRS relationships with cytokine dynamics and drug levels gathered through a review of T-BiSp licensing applications and emerging data from conferences and publications.

Integrated modeling of biomarkers, survival and safety in clinical oncology drug development

Model-based approaches, including population pharmacokinetic-pharmacodynamic modeling, have become an essential component in the clinical phases of oncology drug development. Over the past two decades, models have evolved to describe the temporal dynamics of biomarkers and tumor size, treatment-related adverse events, and their links to survival. Integrated models, defined here as models that incorporate at least two pharmacodynamic/ outcome variables, are applied to answer drug development questions through simulations, e.g., to support the exploration of alternative dosing strategies and study designs in subgroups of patients or other tumor indications. It is expected that these pharmacometric approaches will be expanded as regulatory authorities place further emphasis on early and individualized dosage optimization and inclusive patient-focused development strategies. This review provides an overview of integrated models in the literature, examples of the considerations that need to be made when applying these advanced pharmacometric approaches, and an outlook on the expected further expansion of model-informed drug development of anticancer drugs.

Machine learning models including preoperative and postoperative albumin-bilirubin score: short-term outcomes among patients with hepatocellular carcinoma

BACKGROUND

We sought to assess the impact of various perioperative factors on the risk of severe complications and post-surgical mortality using a novel maching learning technique.

METHODS

Data on patients undergoing resection for HCC were obtained from an international, multi-institutional database between 2000 and 2020. Gradient boosted trees were utilized to construct predictive models.

RESULTS

Among 962 patients who underwent HCC resection, the incidence of severe postoperative complications was 12.7% (n = 122); in-hospital mortality was 2.9% (n = 28). Models that exclusively used preoperative data achieved AUC values of 0.89 (95%CI 0.85 to 0.92) and 0.90 (95%CI 0.84 to 0.96) to predict severe complications and mortality, respectively. Models that combined preoperative and postoperative data achieved AUC values of 0.93 (95%CI 0.91 to 0.96) and 0.92 (95%CI 0.86 to 0.97) for severe morbidity and mortality, respectively. The SHAP algorithm demonstrated that the factor most strongly predictive of severe morbidity and mortality was postoperative day 1 and 3 albumin-bilirubin (ALBI) scores.

CONCLUSION

Incorporation of perioperative data including ALBI scores using ML techniques can help risk-stratify patients undergoing resection of HCC.

Preoperative identification of early extrahepatic recurrence after hepatectomy for colorectal liver metastases: A machine learning approach

BACKGROUND

Machine learning (ML) may provide novel insights into data patterns and improve model prediction accuracy. The current study sought to develop and validate an ML model to predict early extra-hepatic recurrence (EEHR) among patients undergoing resection of colorectal liver metastasis (CRLM).

METHODS

Patients with CRLM who underwent curative-intent resection between 2000 and 2020 were identified from an international multi-institutional database. An eXtreme gradient boosting (XGBoost) model was developed to estimate the risk of EEHR, defined as extrahepatic recurrence within 12 months after hepatectomy, using clinicopathological factors. The relative importance of factors was determined using Shapley additive explanations (SHAP) values.

RESULTS

Among 1410 patients undergoing curative-intent resection, 131 (9.3%) patients experienced EEHR. Median OS among patients with and without EEHR was 35.4 months (interquartile range [IQR] 29.9-46.7) versus 120.5 months (IQR 97.2-134.0), respectively (p < 0.001). The ML predictive model had c-index values of 0.77 (95% CI, 0.72-0.81) and 0.77 (95% CI, 0.73-0.80) in the entire dataset and the validation data set with bootstrapping resamples, respectively. The SHAP algorithm demonstrated that T and N primary tumor categories, as well as tumor burden score were the three most important predictors of EEHR. An easy-to-use risk calculator for EEHR was developed and made available online at: https://junkawashima.shinyapps.io/EEHR/.

CONCLUSIONS

An easy-to-use online calculator was developed using ML to help clinicians predict the chance of EEHR after curative-intent resection for CRLM. This tool may help clinicians in decision-making related to treatment strategies for patients with CRLM.

Impact of an artificial intelligence based model to predict non-transplantable recurrence among patients with hepatocellular carcinoma

OBJECTIVE

We sought to develop Artificial Intelligence (AI) based models to predict non-transplantable recurrence (NTR) of hepatocellular carcinoma (HCC) following hepatic resection (HR).

METHODS

HCC patients who underwent HR between 2000-2020 were identified from a multi-institutional database. NTR was defined as recurrence beyond Milan Criteria. Different machine learning (ML) and deep learning (DL) techniques were used to develop and validate two prediction models for NTR, one using only preoperative factors and a second using both preoperative and postoperative factors.

RESULTS

Overall, 1763 HCC patients were included. Among 877 patients with recurrence, 364 (41.5%) patients developed NTR. An ensemble AI model demonstrated the highest area under ROC curves (AUC) of 0.751 (95% CI: 0.719-0.782) and 0.717 (95% CI:0.653-0.782) in the training and testing cohorts, respectively which improved to 0.858 (95% CI: 0.835-0.884) and 0.764 (95% CI: 0.704-0.826), respectively after incorporation of postoperative pathologic factors. Radiologic tumor burden score and pathological microvascular invasion were the most important preoperative and postoperative factors, respectively to predict NTR. Patients predicted to develop NTR had overall 1- and 5-year survival of 75.6% and 28.2%, versus 93.4% and 55.9%, respectively, among patients predicted to not develop NTR (p < 0.0001).

CONCLUSION

The AI preoperative model may help inform decision of HR versus LT for HCC, while the combined AI model can frame individualized postoperative care (https://altaf-pawlik-hcc-ntr-calculator.streamlit.app/).

Artificial Intelligence: From Buzzword to Useful Tool in Clinical Pharmacology

The advent of artificial intelligence (AI) in clinical pharmacology and drug development is akin to the dawning of a new era. Previously dismissed as merely technological hype, these approaches have emerged as promising tools in different domains, including health care, demonstrating their potential to empower clinical pharmacology decision making, revolutionize the drug development landscape, and advance patient care. Although challenges remain, the remarkable progress already made signals that the leap from hype to reality is well underway, and AI promises to offer clinical pharmacology new tools and possibilities for optimizing patient care is gradually coming to fruition. This review dives into the burgeoning world of AI and machine learning (ML), showcasing different applications of AI in clinical pharmacology and the impact of successful AI/ML implementation on drug development and/or regulatory decisions. This review also highlights recommendations for areas of opportunity in clinical pharmacology, including data analysis (e.g., handling large data sets, screening to identify important covariates, and optimizing patient population) and efficiencies (e.g., automation, translation, literature curation, and training). Realizing the benefits of AI in drug development and understanding its value will lead to the successful integration of AI tools in our clinical pharmacology and pharmacometrics armamentarium.

Artificial Intelligence for Quantitative Modeling in Drug Discovery and Development: An Innovation and Quality Consortium Perspective on Use Cases and Best Practices

Recent breakthroughs in artificial intelligence (AI) and machine learning (ML) have ushered in a new era of possibilities across various scientific domains. One area where these advancements hold significant promise is model-informed drug discovery and development (MID3). To foster a wider adoption and acceptance of these advanced algorithms, the Innovation and Quality (IQ) Consortium initiated the AI/ML working group in 2021 with the aim of promoting their acceptance among the broader scientific community as well as by regulatory agencies. By drawing insights from workshops organized by the working group and attended by key stakeholders across the biopharma industry, academia, and regulatory agencies, this white paper provides a perspective from the IQ Consortium. The range of applications covered in this white paper encompass the following thematic topics: (i) AI/ML-enabled Analytics for Pharmacometrics and Quantitative Systems Pharmacology (QSP) Workflows; (ii) Explainable Artificial Intelligence and its Applications in Disease Progression Modeling; (iii) Natural Language Processing (NLP) in Quantitative Pharmacology Modeling; and (iv) AI/ML Utilization in Drug Discovery. Additionally, the paper offers a set of best practices to ensure an effective and responsible use of AI, including considering the context of use, explainability and generalizability of models, and having human-in-the-loop. We believe that embracing the transformative power of AI in quantitative modeling while adopting a set of good practices can unlock new opportunities for innovation, increase efficiency, and ultimately bring benefits to patients.

Virtual patients, digital twins and causal disease models: paving the ground for in silico clinical trials

Computational models are being explored to simulate in silico the efficacy and safety of drug candidates and medical devices. Disease models that are based on patients' profiling data are being produced to represent interactomes of genes or proteins and to infer causality in the pathophysiology {AuQ: Edit OK?}, which makes it possible to mimic the impact of drugs on relevant targets. Virtual patients designed from medical records as well as digital twins were generated to simulate specific organs and to predict treatment efficacy at the individual patient level {AuQ: Edit OK?}. As the acceptance of digital evidence by regulators grows, predictive artificial intelligence (AI)-based models will support the design of confirmatory trials in humans and will accelerate the development of efficient drugs and medical devices.

Comparison of sequential and joint nonlinear mixed effects modeling of tumor kinetics and survival following Durvalumab treatment in patients with metastatic urothelial carcinoma

Standard endpoints such as objective response rate are usually poorly correlated with overall survival (OS) for treatment with immune checkpoint inhibitors. Longitudinal tumor size may serve as a more useful predictor of OS, and establishing a quantitative relationship between tumor kinetics (TK) and OS is a crucial step for successfully predicting OS based on limited tumor size measurements. This study aims to develop a population TK model in combination with a parametric survival model by sequential and joint modeling approaches to characterize durvalumab phase I/II data from patients with metastatic urothelial cancer, and to evaluate and compare the performance of the two modeling approaches in terms of parameter estimates, TK and survival predictions, and covariate identification. The tumor growth rate constant was estimated to be greater for patients with OS ≤ 16 weeks as compared to that for patients with OS > 16 weeks with the joint modeling approach (k_g= 0.130 vs. 0.0551 week^-1, p-value < 0.0001), but similar for both groups (k_g = 0.0624 vs.0.0563 week^-1, p-value = 0.37) with the sequential modeling approach. The predicted TK profiles by joint modeling appeared better aligned with clinical observations. Joint modeling also predicted OS more accurately than the sequential approach according to concordance index and Brier score. The sequential and joint modeling approaches were also compared using additional simulated datasets, and survival was predicted better by joint modeling in the case of a strong association between TK and OS. In conclusion, joint modeling enabled the establishment of a robust association between TK and OS and may represent a better choice for parametric survival analyses over the sequential approach.

Performance of Cox proportional hazard models on recovering the ground truth of confounded exposure-response relationships for large-molecule oncology drugs

A Cox proportional hazard (CoxPH) model is conventionally used to assess exposure-response (E-R), but its performance to uncover the ground truth when only one dose level of data is available has not been systematically evaluated. We established a simulation workflow to generate realistic E-R datasets to assess the performance of the CoxPH model in recovering the E-R ground truth in various scenarios, considering two potential reasons for the confounded E-R relationship. We found that at high doses, when the pharmacological effects are largely saturated, missing important confounders is the major reason for inferring false-positive E-R relationships. At low doses, when a positive E-R slope is the ground truth, either missing important confounders or mis-specifying the interactions can lead to inaccurate estimates of the E-R slope. This work constructed a simulation workflow generally applicable to clinical datasets to generate clinically relevant simulations and provide an in-depth interpretation on the E-R relationships with confounders inferred by the conventional CoxPH model.

A novel analytical framework for risk stratification of real-world data using machine learning: A small cell lung cancer study

In recent studies, small cell lung cancer (SCLC) treatment guidelines based on Veterans' Administration Lung Study Group limited/extensive disease staging and resulted in broad and inseparable prognostic subgroups. Evidence suggests that the eight versions of tumor, node, and metastasis (TNM) staging can play an important role to address this issue. The aim of the present study was to improve the detection of prognostic subgroups from a real-word data (RWD) cohort of patients and analyze their patterns using a development pipeline with thoracic oncologists and machine learning methods. The method detected subgroups of patients informing unsupervised learning (partition around medoids) including the impact of covariates on prognosis (Cox regression and random survival forest). An analysis was carried out using patients with SCLC (n = 636) with stage IIIA-IVB according to TNM classification. The analysis yielded k = 7 compacted and well-separated clusters of patients. Performance status (Eastern Cooperative Oncology Group-Performance Status), lactate dehydrogenase, spreading of metastasis, cancer stage, and CRP were the baselines that characterized the subgroups. The selected clustering method outperformed standard clustering techniques, which were not capable of detecting meaningful subgroups. From the analysis of cluster treatment decisions, we showed the potential of future RWD applications to understand disease, develop individualized therapies, and improve healthcare decision making.

Neural Networks for Survival Prediction in Medicine Using Prognostic Factors: A Review and Critical Appraisal

Survival analysis deals with the expected duration of time until one or more events of interest occur. Time to the event of interest may be unobserved, a phenomenon commonly known as right censoring, which renders the analysis of these data challenging. Over the years, machine learning algorithms have been developed and adapted to right-censored data. Neural networks have been repeatedly employed to build clinical prediction models in healthcare with a focus on cancer and cardiology. We present the first ever attempt at a large-scale review of survival neural networks (SNNs) with prognostic factors for clinical prediction in medicine. This work provides a comprehensive understanding of the literature (24 studies from 1990 to August 2021, global search in PubMed). Relevant manuscripts are classified as methodological/technical (novel methodology or new theoretical model; 13 studies) or applications (11 studies). We investigate how researchers have used neural networks to fit survival data for prediction. There are two methodological trends: either time is added as part of the input features and a single output node is specified, or multiple output nodes are defined for each time interval. A critical appraisal of model aspects that should be designed and reported more carefully is performed. We identify key characteristics of prediction models (i.e., number of patients/predictors, evaluation measures, calibration), and compare ANN's predictive performance to the Cox proportional hazards model. The median sample size is 920 patients, and the median number of predictors is 7. Major findings include poor reporting (e.g., regarding missing data, hyperparameters) as well as inaccurate model development/validation. Calibration is neglected in more than half of the studies. Cox models are not developed to their full potential and claims for the performance of SNNs are exaggerated. Light is shed on the current state of art of SNNs in medicine with prognostic factors. Recommendations are made for the reporting of clinical prediction models. Limitations are discussed, and future directions are proposed for researchers who seek to develop existing methodology.

Individual risk prediction: Comparing random forests with Cox proportional-hazards model by a simulation study

With big data becoming widely available in healthcare, machine learning algorithms such as random forest (RF) that ignores time-to-event information and random survival forest (RSF) that handles right-censored data are used for individual risk prediction alternatively to the Cox proportional hazards (Cox-PH) model. We aimed to systematically compare RF and RSF with Cox-PH. RSF with three split criteria [log-rank (RSF-LR), log-rank score (RSF-LRS), maximally selected rank statistics (RSF-MSR)]; RF, Cox-PH, and Cox-PH with splines (Cox-S) were evaluated through a simulation study based on real data. One hundred eighty scenarios were investigated assuming different associations between the predictors and the outcome (linear/linear and interactions/nonlinear/nonlinear and interactions), training sample sizes (500/1000/5000), censoring rates (50%/75%/93%), hazard functions (increasing/decreasing/constant), and number of predictors (seven, 15 including noise variables). Methods' performance was evaluated with time-dependent area under curve and integrated Brier score. In all scenarios, RF had the worst performance. In scenarios with a low number of events (⩽70), Cox-PH was at least noninferior to RSF, whereas under linearity assumption it outperformed RSF. Under the presence of interactions, RSF performed better than Cox-PH as the number of events increased whereas Cox-S reached at least similar performance with RSF under nonlinear effects. RSF-LRS performed slightly worse than RSF-LR and RSF-MSR when including noise variables and interaction effects. When applied to real data, models incorporating survival time performed better. Although RSF algorithms are a promising alternative to conventional Cox-PH as data complexity increases, they require a higher number of events for training. In time-to-event analysis, algorithms that consider survival time should be used.

Adoption of Machine Learning in Pharmacometrics: An Overview of Recent Implementations and Their Considerations

Pharmacometrics is a multidisciplinary field utilizing mathematical models of physiology, pharmacology, and disease to describe and quantify the interactions between medication and patient. As these models become more and more advanced, the need for advanced data analysis tools grows. Recently, there has been much interest in the adoption of machine learning (ML) algorithms. These algorithms offer strong function approximation capabilities and might reduce the time spent on model development. However, ML tools are not yet an integral part of the pharmacometrics workflow. The goal of this work is to discuss how ML algorithms have been applied in four stages of the pharmacometrics pipeline: data preparation, hypothesis generation, predictive modelling, and model validation. We will also discuss considerations before the use of ML algorithms with respect to each topic. We conclude by summarizing applications that hold potential for adoption by pharmacometricians.

Decoding kinase-adverse event associations for small molecule kinase inhibitors

Small molecule kinase inhibitors (SMKIs) are being approved at a fast pace under expedited programs for anticancer treatment. In this study, we construct a multi-domain dataset from a total of 4638 patients in the registrational trials of 16 FDA-approved SMKIs and employ a machine-learning model to examine the relationships between kinase targets and adverse events (AEs). Internal and external (datasets from two independent SMKIs) validations have been conducted to verify the usefulness of the established model. We systematically evaluate the potential associations between 442 kinases with 2145 AEs and made publicly accessible an interactive web application “Identification of Kinase-Specific Signal” (https://gongj.shinyapps.io/ml4ki). The developed model (1) provides a platform for experimentalists to identify and verify undiscovered KI-AE pairs, (2) serves as a precision-medicine tool to mitigate individual patient safety risks by forecasting clinical safety signals and (3) can function as a modern drug development tool to screen and compare SMKI target therapies from the safety perspective.

Small molecule kinase inhibitors (SMKIs) are being approved at a fast pace under expedited programs for anticancer treatment. Here, the authors employ a machine-learning model to examine the relationships between kinase targets and adverse events in the trials of 16 FDA-approved SMKIs.

A scoping methodological review of simulation studies comparing statistical and machine learning approaches to risk prediction for time-to-event data

BACKGROUND

There is substantial interest in the adaptation and application of so-called machine learning approaches to prognostic modelling of censored time-to-event data. These methods must be compared and evaluated against existing methods in a variety of scenarios to determine their predictive performance. A scoping review of how machine learning methods have been compared to traditional survival models is important to identify the comparisons that have been made and issues where they are lacking, biased towards one approach or misleading.

METHODS

We conducted a scoping review of research articles published between 1 January 2000 and 2 December 2020 using PubMed. Eligible articles were those that used simulation studies to compare statistical and machine learning methods for risk prediction with a time-to-event outcome in a medical/healthcare setting. We focus on data-generating mechanisms (DGMs), the methods that have been compared, the estimands of the simulation studies, and the performance measures used to evaluate them.

RESULTS

A total of ten articles were identified as eligible for the review. Six of the articles evaluated a method that was developed by the authors, four of which were machine learning methods, and the results almost always stated that this developed method's performance was equivalent to or better than the other methods compared. Comparisons were often biased towards the novel approach, with the majority only comparing against a basic Cox proportional hazards model, and in scenarios where it is clear it would not perform well. In many of the articles reviewed, key information was unclear, such as the number of simulation repetitions and how performance measures were calculated.

CONCLUSION

It is vital that method comparisons are unbiased and comprehensive, and this should be the goal even if realising it is difficult. Fully assessing how newly developed methods perform and how they compare to a variety of traditional statistical methods for prognostic modelling is imperative as these methods are already being applied in clinical contexts. Evaluations of the performance and usefulness of recently developed methods for risk prediction should be continued and reporting standards improved as these methods become increasingly popular.

Application of machine learning based methods in exposure-response analysis

Robust estimation of exposure response analysis relies on correct specification of the model structure with traditional parametric approach. However, the assumptions of the handcrafted model may not always hold or verifiable. Here, we conducted a simulation study to assess the performance of machine learning-based techniques in exposure-response (E-R) analysis where data were generated by a complicated nonlinear system under one dose level. Two analysis options involving machine learning were evaluated. The first option was based on marginal structural model with inverse probability weighting, where machine learning (ML) was employed to improve the performance of propensity score estimation. The simulation results showed that propensity score predicted by ML was more robust than traditional multinomial logistic regression in terms of adjusting the confounding effects and unbiasedly estimating the E-R relationship. The second option estimated the E-R relationship by employing artificial neural network as a universal function approximator to the data generating mechanism, without the requirement of accurately hand-crafting the whole simulation system. The results demonstrated that the trained network was able to correctly predict the treatment effects across a certain range of adjacent dose levels. In contrast, traditional regression provided biased predictions, even when all confounders were included in the model. Our study demonstrated that ML may serve as a powerful tool for pharmacometrics analysis with its prediction flexibility in a nonlinear system and its capacity of approximating the ground truth.

A Simulation Study to Compare the Predictive Performance of Survival Neural Networks with Cox Models for Clinical Trial Data

Background

Studies focusing on prediction models are widespread in medicine. There is a trend in applying machine learning (ML) by medical researchers and clinicians. Over the years, multiple ML algorithms have been adapted to censored data. However, the choice of methodology should be motivated by the real-life data and their complexity. Here, the predictive performance of ML techniques is compared with statistical models in a simple clinical setting (small/moderate sample size and small number of predictors) with Monte-Carlo simulations.

Methods

Synthetic data (250 or 1000 patients) were generated that closely resembled 5 prognostic factors preselected based on a European Osteosarcoma Intergroup study (MRC BO06/EORTC 80931). Comparison was performed between 2 partial logistic artificial neural networks (PLANNs) and Cox models for 20, 40, 61, and 80% censoring. Survival times were generated from a log-normal distribution. Models were contrasted in terms of the C-index, Brier score at 0-5 years, integrated Brier score (IBS) at 5 years, and miscalibration at 2 and 5 years (usually neglected). The endpoint of interest was overall survival.

Results

PLANNs original/extended were tuned based on the IBS at 5 years and the C-index, achieving a slightly better performance with the IBS. Comparison with Cox models showed that PLANNs can reach similar predictive performance on simulated data for most scenarios with respect to the C-index, Brier score, or IBS. However, Cox models were frequently less miscalibrated. Performance was robust in scenario data where censored patients were removed before 2 years or curtailing at 5 years was performed (on training data).

Conclusion

Survival neural networks reached a comparable predictive performance with Cox models but were generally less well calibrated. All in all, researchers should be aware of burdensome aspects of ML techniques such as data preprocessing, tuning of hyperparameters, and computational intensity that render them disadvantageous against conventional regression models in a simple clinical setting.

Random survival forests for dynamic predictions of a time-to-event outcome using a longitudinal biomarker

BACKGROUND

Risk prediction models for time-to-event outcomes play a vital role in personalized decision-making. A patient's biomarker values, such as medical lab results, are often measured over time but traditional prediction models ignore their longitudinal nature, using only baseline information. Dynamic prediction incorporates longitudinal information to produce updated survival predictions during follow-up. Existing methods for dynamic prediction include joint modeling, which often suffers from computational complexity and poor performance under misspecification, and landmarking, which has a straightforward implementation but typically relies on a proportional hazards model. Random survival forests (RSF), a machine learning algorithm for time-to-event outcomes, can capture complex relationships between the predictors and survival without requiring prior specification and has been shown to have superior predictive performance.

METHODS

We propose an alternative approach for dynamic prediction using random survival forests in a landmarking framework. With a simulation study, we compared the predictive performance of our proposed method with Cox landmarking and joint modeling in situations where the proportional hazards assumption does not hold and the longitudinal marker(s) have a complex relationship with the survival outcome. We illustrated the use of the RSF landmark approach in two clinical applications to assess the performance of various RSF model building decisions and to demonstrate its use in obtaining dynamic predictions.

RESULTS

In simulation studies, RSF landmarking outperformed joint modeling and Cox landmarking when a complex relationship between the survival and longitudinal marker processes was present. It was also useful in application when there were several predictors for which the clinical relevance was unknown and multiple longitudinal biomarkers were present. Individualized dynamic predictions can be obtained from this method and the variable importance metric is useful for examining the changing predictive power of variables over time. In addition, RSF landmarking is easily implementable in standard software and using suggested specifications requires less computation time than joint modeling.

CONCLUSIONS

RSF landmarking is a nonparametric, machine learning alternative to current methods for obtaining dynamic predictions when there are complex or unknown relationships present. It requires little upfront decision-making and has comparable predictive performance and has preferable computational speed.

Application of Deep Neural Networks as a Prescreening Tool to Assign Individualized Absorption Models in Pharmacokinetic Analysis

A specific model for drug absorption is necessarily assumed in pharmacokinetic (PK) analyses following extravascular dosing. Unfortunately, an inappropriate absorption model may force other model parameters to be poorly estimated. An added complexity arises in population PK analyses when different individuals appear to have different absorption patterns. The aim of this study is to demonstrate that a deep neural network (DNN) can be used to prescreen data and assign an individualized absorption model consistent with either a first-order, Erlang, or split-peak process. Ten thousand profiles were simulated for each of the three aforementioned shapes and used for training the DNN algorithm with a 30% hold-out validation set. During the training phase, a 99.7% accuracy was attained, with 99.4% accuracy during in the validation process. In testing the algorithm classification performance with external patient data, a 93.7% accuracy was reached. This algorithm was developed to prescreen individual data and assign a particular absorption model prior to a population PK analysis. We envision it being used as an efficient prescreening tool in other situations that involve a model component that appears to be variable across subjects. It has the potential to reduce the time needed to perform a manual visual assignment and eliminate inter-assessor variability and bias in assigning a sub-model.

Fast screening of covariates in population models empowered by machine learning

One of the objectives of Pharmacometry (PMX) population modeling is the identification of significant and clinically relevant relationships between parameters and covariates. Here, we demonstrate how this complex selection task could benefit from supervised learning algorithms using importance scores. We compare various classical methods with three machine learning (ML) methods applied to NONMEM empirical Bayes estimates: random forest, neural networks (NNs), and support vector regression (SVR). The performance of the ML models is assessed using receiver operating characteristic (ROC) curves. The F1 score, which measures test accuracy, is used to compare ML and PMX approaches. Methods are applied to different scenarios of covariate influence based on simulated pharmacokinetics data. ML achieved similar or better F1 scores than stepwise covariate modeling (SCM) and conditional sampling for stepwise approach based on correlation tests (COSSAC). Correlations between covariates and the number of false covariates does not affect the performance of any method, but effect size has an impact. Methods are not equivalent with respect to computational speed; SCM is 30 and 100-times slower than NN and SVR, respectively. The results are validated in an additional scenario involving 100 covariates. Taken together, the results indicate that ML methods can greatly increase the efficiency of population covariate model building in the case of large datasets or complex models that require long run-times. This can provide fast initial covariate screening, which can be followed by more conventional PMX approaches to assess the clinical relevance of selected covariates and build the final model.

Application of Machine Learning in Translational Medicine: Current Status and Future Opportunities

The exponential increase in our ability to harness multi-dimensional biological and clinical data from experimental to real-world settings has transformed pharmaceutical research and development in recent years, with increasing applications of artificial intelligence (AI) and machine learning (ML). Patient-centered iterative forward and reverse translation is at the heart of precision medicine discovery and development across the continuum from target validation to optimization of pharmacotherapy. Integration of advanced analytics into the practice of Translational Medicine is now a fundamental enabler to fully exploit information contained in diverse sources of big data sets such as “omics” data, as illustrated by deep characterizations of the genome, transcriptome, proteome, metabolome, microbiome, and exposome. In this commentary, we provide an overview of ML applications in drug discovery and development, aligned with the three strategic pillars of Translational Medicine (target, patient, dose) and offer perspectives on their potential to transform the science and practice of the discipline. Opportunities for integrating ML approaches into the discipline of Pharmacometrics are discussed and will revolutionize the practice of model-informed drug discovery and development. Finally, we posit that joint efforts of Clinical Pharmacology, Bioinformatics, and Biomarker Technology experts are vital in cross-functional team settings to realize the promise of AI/ML-enabled Translational and Precision Medicine.

Early predictions of response and survival from a tumor dynamics model in patients with recurrent, metastatic head and neck squamous cell carcinoma treated with immunotherapy

We developed and evaluated a method for making early predictions of best overall response (BOR) and overall survival at 6 months (OS6) in patients with cancer treated with immunotherapy. This method combines machine learning with modeling of longitudinal tumor size data. We applied our method to data from durvalumab‐exposed patients with recurrent/metastatic head and neck cancer. A fivefold cross‐validation was used for model selection. Independent trial data, with various degrees of data truncation, were used for model validation. Mean classification error rates (90% confidence intervals [CIs]) from cross‐validation were 5.99% (90% CI 2.98%–7.50%) for BOR and 19.8% (90% CI 15.8%–39.3%) for OS6. During model validation, the area under the receiver operating characteristic curves was preserved for BOR (0.97, 0.97, and 0.94) and OS6 (0.85, 0.84, and 0.82) at 24, 18, and 12 weeks, respectively. These results suggest our method predicts trial outcomes accurately from early data and could be used to aid drug development.

Artificial Intelligence and Machine Learning Applied at the Point of Care

Introduction

The increasing availability of healthcare data and rapid development of big data analytic methods has opened new avenues for use of Artificial Intelligence (AI)- and Machine Learning (ML)-based technology in medical practice. However, applications at the point of care are still scarce.

Objective

Review and discuss case studies to understand current capabilities for applying AI/ML in the healthcare setting, and regulatory requirements in the US, Europe and China.

Methods

A targeted narrative literature review of AI/ML based digital tools was performed. Scientific publications (identified in PubMed) and grey literature (identified on the websites of regulatory agencies) were reviewed and analyzed.

Results

From the regulatory perspective, AI/ML-based solutions can be considered medical devices (i.e., Software as Medical Device, SaMD). A case series of SaMD is presented. First, tools for monitoring and remote management of chronic diseases are presented. Second, imaging applications for diagnostic support are discussed. Finally, clinical decision support tools to facilitate the choice of treatment and precision dosing are reviewed. While tested and validated algorithms for precision dosing exist, their implementation at the point of care is limited, and their regulatory and commercialization pathway is not clear. Regulatory requirements depend on the level of risk associated with the use of the device in medical practice, and can be classified into administrative (manufacturing and quality control), software-related (design, specification, hazard analysis, architecture, traceability, software risk analysis, cybersecurity, etc.), clinical evidence (including patient perspectives in some cases), non-clinical evidence (dosing validation and biocompatibility/toxicology) and other, such as e.g. benefit-to-risk determination, risk assessment and mitigation. There generally is an alignment between the US and Europe. China additionally requires that the clinical evidence is applicable to the Chinese population and recommends that a third-party central laboratory evaluates the clinical trial results.

Conclusions

The number of promising AI/ML-based technologies is increasing, but few have been implemented widely at the point of care. The need for external validation, implementation logistics, and data exchange and privacy remain the main obstacles.

Predictive Analysis of First Abbreviated New Drug Application Submission for New Chemical Entities Based on Machine Learning Methodology

Generic drug products are approved by the US Food and Drug Administration (FDA) through Abbreviated New Drug Applications (ANDAs). The ANDA review and approval involves multiple offices across the FDA. Forecasting ANDA submissions can critically inform resource allocation and workload management. In this work, we used machine learning (ML) methodologies to predict the time to first ANDA submissions referencing new chemical entities following their earliest lawful ANDA submission dates. Drug product information, regulatory factors, and pharmacoeconomic factors were used as modeling inputs. The random survival forest ML method, as well as the conventional Cox model, was used for ANDA submission predictions. The ML method outperformed the conventional Cox regression model in predictive performance that was adequately assessed by both internal and external validations. In conclusion, it can potentially serve as an effective forecasting tool for strategic workload and research planning for generic applications.