1
|
Ayyar VS, Jusko WJ. Transitioning from Basic toward Systems Pharmacodynamic Models: Lessons from Corticosteroids. Pharmacol Rev 2020; 72:414-438. [PMID: 32123034 DOI: 10.1124/pr.119.018101] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023] Open
Abstract
Technology in bioanalysis, -omics, and computation have evolved over the past half century to allow for comprehensive assessments of the molecular to whole body pharmacology of diverse corticosteroids. Such studies have advanced pharmacokinetic and pharmacodynamic (PK/PD) concepts and models that often generalize across various classes of drugs. These models encompass the "pillars" of pharmacology, namely PK and target drug exposure, the mass-law interactions of drugs with receptors/targets, and the consequent turnover and homeostatic control of genes, biomarkers, physiologic responses, and disease symptoms. Pharmacokinetic methodology utilizes noncompartmental, compartmental, reversible, physiologic [full physiologically based pharmacokinetic (PBPK) and minimal PBPK], and target-mediated drug disposition models using a growing array of pharmacometric considerations and software. Basic PK/PD models have emerged (simple direct, biophase, slow receptor binding, indirect response, irreversible, turnover with inactivation, and transduction models) that place emphasis on parsimony, are mechanistic in nature, and serve as highly useful "top-down" methods of quantitating the actions of diverse drugs. These are often components of more complex quantitative systems pharmacology (QSP) models that explain the array of responses to various drugs, including corticosteroids. Progressively deeper mechanistic appreciation of PBPK, drug-target interactions, and systems physiology from the molecular (genomic, proteomic, metabolomic) to cellular to whole body levels provides the foundation for enhanced PK/PD to comprehensive QSP models. Our research based on cell, animal, clinical, and theoretical studies with corticosteroids have provided ideas and quantitative methods that have broadly advanced the fields of PK/PD and QSP modeling and illustrates the transition toward a global, systems understanding of actions of diverse drugs. SIGNIFICANCE STATEMENT: Over the past half century, pharmacokinetics (PK) and pharmacokinetics/pharmacodynamics (PK/PD) have evolved to provide an array of mechanism-based models that help quantitate the disposition and actions of most drugs. We describe how many basic PK and PK/PD model components were identified and often applied to the diverse properties of corticosteroids (CS). The CS have complications in disposition and a wide array of simple receptor-to complex gene-mediated actions in multiple organs. Continued assessments of such complexities have offered opportunities to develop models ranging from simple PK to enhanced PK/PD to quantitative systems pharmacology (QSP) that help explain therapeutic and adverse CS effects. Concurrent development of state-of-the-art PK, PK/PD, and QSP models are described alongside experimental studies that revealed diverse CS actions.
Collapse
Affiliation(s)
- Vivaswath S Ayyar
- Department of Pharmaceutical Sciences University at Buffalo, School of Pharmacy and Pharmaceutical Sciences, Buffalo, New York
| | - William J Jusko
- Department of Pharmaceutical Sciences University at Buffalo, School of Pharmacy and Pharmaceutical Sciences, Buffalo, New York
| |
Collapse
|
2
|
Metz DK, Holford N, Kausman JY, Walker A, Cranswick N, Staatz CE, Barraclough KA, Ierino F. Optimizing Mycophenolic Acid Exposure in Kidney Transplant Recipients: Time for Target Concentration Intervention. Transplantation 2019; 103:2012-2030. [PMID: 31584924 PMCID: PMC6756255 DOI: 10.1097/tp.0000000000002762] [Citation(s) in RCA: 41] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2019] [Revised: 03/29/2019] [Accepted: 04/03/2019] [Indexed: 12/24/2022]
Abstract
The immunosuppressive agent mycophenolate is used extensively in kidney transplantation, yet dosing strategy applied varies markedly from fixed dosing ("one-dose-fits-all"), to mycophenolic acid (MPA) trough concentration monitoring, to dose optimization to an MPA exposure target (as area under the concentration-time curve [MPA AUC0-12]). This relates in part to inconsistent results in prospective trials of concentration-controlled dosing (CCD). In this review, the totality of evidence supporting mycophenolate CCD is examined: pharmacological characteristics, observational data linking exposure to efficacy and toxicities, and randomized controlled trials of CCD, with attention to dose optimization method and exposure achieved. Fixed dosing of mycophenolate consistently leads to underexposure associated with rejection, as well as overexposure associated with toxicities. When CCD is driven by pharmacokinetic calculation to a target concentration (target concentration intervention), MPA exposure is successfully controlled and clinical benefits are seen. There remains a need for consensus on practical aspects of mycophenolate target concentration intervention in contemporary tacrolimus-containing regimens and future research to define maintenance phase exposure targets. However, given ongoing consequences of both overimmunosuppression and underimmunosuppression in kidney transplantation, impacting short- and long-term outcomes, these should be a priority. The imprecise "one-dose-fits-all" approach should be replaced by the clinically proven MPA target concentration strategy.
Collapse
Affiliation(s)
- David K. Metz
- Department of Nephrology, Royal Children’s Hospital, Melbourne, VIC, Australia
- Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia
- Murdoch Children’s Research Institute, Melbourne, VIC, Australia
- Clinical Pharmacology Unit, Royal Children’s Hospital, Melbourne, VIC, Australia
| | - Nick Holford
- Department of Pharmacology and Clinical Pharmacology, University of Auckland, Auckland, New Zealand
| | - Joshua Y. Kausman
- Department of Nephrology, Royal Children’s Hospital, Melbourne, VIC, Australia
- Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia
- Murdoch Children’s Research Institute, Melbourne, VIC, Australia
| | - Amanda Walker
- Department of Nephrology, Royal Children’s Hospital, Melbourne, VIC, Australia
- Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia
- Murdoch Children’s Research Institute, Melbourne, VIC, Australia
| | - Noel Cranswick
- Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia
- Murdoch Children’s Research Institute, Melbourne, VIC, Australia
- Clinical Pharmacology Unit, Royal Children’s Hospital, Melbourne, VIC, Australia
| | | | - Katherine A. Barraclough
- Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia
- Department of Nephrology, Royal Melbourne Hospital, Melbourne, VIC, Australia
| | - Francesco Ierino
- Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia
- Department of Nephrology, St Vincent’s Health, Melbourne, VIC, Australia
| |
Collapse
|
3
|
Gal J, Milano G, Ferrero JM, Saâda-Bouzid E, Viotti J, Chabaud S, Gougis P, Le Tourneau C, Schiappa R, Paquet A, Chamorey E. Optimizing drug development in oncology by clinical trial simulation: Why and how? Brief Bioinform 2019; 19:1203-1217. [PMID: 28575140 DOI: 10.1093/bib/bbx055] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2017] [Indexed: 12/11/2022] Open
Abstract
In therapeutic research, the safety and efficacy of pharmaceutical products are necessarily tested on humans via clinical trials after an extensive and expensive preclinical development period. Methodologies such as computer modeling and clinical trial simulation (CTS) might represent a valuable option to reduce animal and human assays. The relevance of these methods is well recognized in pharmacokinetics and pharmacodynamics from the preclinical phase to postmarketing. However, they are barely used and are poorly regarded for drug approval, despite Food and Drug Administration and European Medicines Agency recommendations. The generalization of CTS could be greatly facilitated by the availability of software for modeling biological systems, by clinical trial studies and hospital databases. Data sharing and data merging raise legal, policy and technical issues that will need to be addressed. Development of future molecules will have to use CTS for faster development and thus enable better patient management. Drug activity modeling coupled with disease modeling, optimal use of medical data and increased computing speed should allow this leap forward. The realization of CTS requires not only bioinformatics tools to allow interconnection and global integration of all clinical data but also a universal legal framework to protect the privacy of every patient. While recognizing that CTS can never replace 'real-life' trials, they should be implemented in future drug development schemes to provide quantitative support for decision-making. This in silico medicine opens the way to the P4 medicine: predictive, preventive, personalized and participatory.
Collapse
Affiliation(s)
- Jocelyn Gal
- Epidemiology and Biostatistics Unit at the Antoine Lacassagne Center, Nice, France
| | | | | | | | | | | | - Paul Gougis
- Pitie´-Salp^etrie`re Hospital in Paris, France
| | | | | | - Agnes Paquet
- Molecular and Cellular Pharmacology Institute of Sophia Antipolis, Valbonne, France
| | | |
Collapse
|
4
|
Seng Yue C, Ozdin D, Selber-Hnatiw S, Ducharme MP. Opportunities and Challenges Related to the Implementation of Model-Based Bioequivalence Criteria. Clin Pharmacol Ther 2019; 105:350-362. [PMID: 30375647 DOI: 10.1002/cpt.1270] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2018] [Accepted: 10/12/2018] [Indexed: 11/09/2022]
Abstract
The science of bioequivalence and biosimilarity has greatly evolved over the past 3 decades. Current methods for assessing bioequivalence mostly rely on noncompartmental pharmacokinetic (PK) analyses, which have proven to be reliable and robust for most products. However, the development of more complex products is forcing scientists and regulators to consider alternative approaches, including those derived from model-based population PK analyses. This article will examine the strengths and weaknesses of standard noncompartmental methods and compare them to model-based approaches, including a comparison of metrics associated with each method. Specific situations for which model-based approaches could prove to be more suitable will be presented, as well as potential bioequivalence metrics that could be considered for bioequivalence comparisons. The opportunities and challenges that are associated with these novel methods will also be discussed.
Collapse
Affiliation(s)
| | - Deniz Ozdin
- Learn and Confirm, Inc., St-Laurent, Quebec, Canada.,Faculté de Pharmacie, University of Montreal, Montreal, Quebec, Canada
| | | | - Murray P Ducharme
- Learn and Confirm, Inc., St-Laurent, Quebec, Canada.,Faculté de Pharmacie, University of Montreal, Montreal, Quebec, Canada
| |
Collapse
|
5
|
Bloomingdale P, Housand C, Apgar JF, Millard BL, Mager DE, Burke JM, Shah DK. Quantitative systems toxicology. CURRENT OPINION IN TOXICOLOGY 2017; 4:79-87. [PMID: 29308440 PMCID: PMC5754001 DOI: 10.1016/j.cotox.2017.07.003] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
The overarching goal of modern drug development is to optimize therapeutic benefits while minimizing adverse effects. However, inadequate efficacy and safety concerns remain to be the major causes of drug attrition in clinical development. For the past 80 years, toxicity testing has consisted of evaluating the adverse effects of drugs in animals to predict human health risks. The U.S. Environmental Protection Agency recognized the need to develop innovative toxicity testing strategies and asked the National Research Council to develop a long-range vision and strategy for toxicity testing in the 21st century. The vision aims to reduce the use of animals and drug development costs through the integration of computational modeling and in vitro experimental methods that evaluates the perturbation of toxicity-related pathways. Towards this vision, collaborative quantitative systems pharmacology and toxicology modeling endeavors (QSP/QST) have been initiated amongst numerous organizations worldwide. In this article, we discuss how quantitative structure-activity relationship (QSAR), network-based, and pharmacokinetic/pharmacodynamic modeling approaches can be integrated into the framework of QST models. Additionally, we review the application of QST models to predict cardiotoxicity and hepatotoxicity of drugs throughout their development. Cell and organ specific QST models are likely to become an essential component of modern toxicity testing, and provides a solid foundation towards determining individualized therapeutic windows to improve patient safety.
Collapse
Affiliation(s)
- Peter Bloomingdale
- Department of Pharmaceutical Sciences, School of Pharmacy and Pharmaceutical Sciences, The State University of New York at Buffalo, Buffalo, NY, USA
| | - Conrad Housand
- Applied BioMath, LLC, 55 Old Bedford Road, Suite 208, Lincoln, MA 01773, USA
| | - Joshua F Apgar
- Applied BioMath, LLC, 55 Old Bedford Road, Suite 208, Lincoln, MA 01773, USA
| | - Bjorn L Millard
- Applied BioMath, LLC, 55 Old Bedford Road, Suite 208, Lincoln, MA 01773, USA
| | - Donald E Mager
- Department of Pharmaceutical Sciences, School of Pharmacy and Pharmaceutical Sciences, The State University of New York at Buffalo, Buffalo, NY, USA
| | - John M Burke
- Applied BioMath, LLC, 55 Old Bedford Road, Suite 208, Lincoln, MA 01773, USA
| | - Dhaval K Shah
- Department of Pharmaceutical Sciences, School of Pharmacy and Pharmaceutical Sciences, The State University of New York at Buffalo, Buffalo, NY, USA
| |
Collapse
|
6
|
Wickremsinhe ER, Renninger M, Paulman A, Pritt M, Schultze AE. Impact of Repeated Tail Clip and Saphenous Vein Phlebotomy on Rats Used in Toxicology Studies. Toxicol Pathol 2016; 44:1013-20. [DOI: 10.1177/0192623316656285] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
Abstract
Sampling blood for toxicokinetic (TK) evaluation in rodents is typically performed using a satellite group of animals to avoid depleting the blood volume and inducing an additional stressor in the main study animals. This practice does not allow for direct comparison of individual animal toxicity to exposure. These studies evaluated serial collection of twelve, 40-µl blood samples from each rat from either a tail clip or a saphenous vein bleed and its impact on toxicologic parameters over 4- and 14-day periods. The results show the feasibility of successfully collecting TK samples from main study animals, using either of the two techniques. Both procedures were amenable to execution by a single technician using dried blood spot sampling. Any changes observed in the primary markers of erythroid mass between the nonbled control rats and repeat sampled rats were minimal and the range of values often overlapped. This technique would improve the quality of data generated from toxicology studies by allowing a direct comparison of systemic exposure to toxicity while at the same time reducing the number of rats by obviating the need for satellite groups.
Collapse
Affiliation(s)
| | | | | | - Michael Pritt
- Lilly Research Laboratories, Indianapolis, Indiana, USA
| | | |
Collapse
|
7
|
Kimko H, Pinheiro J. Model-based clinical drug development in the past, present and future: a commentary. Br J Clin Pharmacol 2015; 79:108-16. [PMID: 24527997 DOI: 10.1111/bcp.12341] [Citation(s) in RCA: 39] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2013] [Accepted: 01/27/2014] [Indexed: 02/05/2023] Open
Abstract
Clinical drug development remains a mostly empirical, costly enterprise, in which decision-making is often based on qualitative assessment of risk, without properly leveraging all the relevant data collected throughout the development programme. Model-based drug development (MBDD) has been proposed by regulatory agencies, academia and pharmaceutical companies as a paradigm to modernize drug research through the quantification of risk and combination of information from different sources across time. We present here a historical account of the use of MBDD in clinical drug development, the current challenges and further opportunities for its application in the pharmaceutical industry.
Collapse
Affiliation(s)
- Holly Kimko
- Model Based Drug Development, Janssen Research & Development, LLC of Johnson & Johnson, Raritan, NJ, 08869, USA
| | | |
Collapse
|
8
|
Ng CM. Novel hybrid GPU-CPU implementation of parallelized Monte Carlo parametric expectation maximization estimation method for population pharmacokinetic data analysis. AAPS JOURNAL 2013; 15:1212-21. [PMID: 24002801 DOI: 10.1208/s12248-013-9524-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/10/2012] [Accepted: 08/07/2013] [Indexed: 11/30/2022]
Abstract
The development of a population PK/PD model, an essential component for model-based drug development, is both time- and labor-intensive. A graphical-processing unit (GPU) computing technology has been proposed and used to accelerate many scientific computations. The objective of this study was to develop a hybrid GPU-CPU implementation of parallelized Monte Carlo parametric expectation maximization (MCPEM) estimation algorithm for population PK data analysis. A hybrid GPU-CPU implementation of the MCPEM algorithm (MCPEMGPU) and identical algorithm that is designed for the single CPU (MCPEMCPU) were developed using MATLAB in a single computer equipped with dual Xeon 6-Core E5690 CPU and a NVIDIA Tesla C2070 GPU parallel computing card that contained 448 stream processors. Two different PK models with rich/sparse sampling design schemes were used to simulate population data in assessing the performance of MCPEMCPU and MCPEMGPU. Results were analyzed by comparing the parameter estimation and model computation times. Speedup factor was used to assess the relative benefit of parallelized MCPEMGPU over MCPEMCPU in shortening model computation time. The MCPEMGPU consistently achieved shorter computation time than the MCPEMCPU and can offer more than 48-fold speedup using a single GPU card. The novel hybrid GPU-CPU implementation of parallelized MCPEM algorithm developed in this study holds a great promise in serving as the core for the next-generation of modeling software for population PK/PD analysis.
Collapse
Affiliation(s)
- C M Ng
- Division of Clinical Pharmacology and Therapeutics, Children's Hospital of Philadelphia, CTRB Building Room 4010, 3501 Civic Center Blvd, Philadelphia, Pennsylvania, 19104, USA,
| |
Collapse
|
9
|
Mudd PN, Groenendaal H, Bush MA, Schmith VD. Probabilistic Risk Analysis: Improving Early Drug Development Decision Making. Clin Pharmacol Ther 2010; 88:871-5. [DOI: 10.1038/clpt.2010.231] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
10
|
Metabotropic glutamate receptor modulation, translational methods, and biomarkers: relationships with anxiety. Psychopharmacology (Berl) 2008; 199:389-402. [PMID: 18322676 DOI: 10.1007/s00213-008-1096-9] [Citation(s) in RCA: 25] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/23/2007] [Accepted: 01/28/2008] [Indexed: 01/31/2023]
Abstract
RATIONALE The increasing awareness of the need to align clinical and preclinical research to facilitate rapid development of new drug therapies is reflected in the recent introduction of the term "translational medicine". This review examines the implications of translational medicine for psychiatric disorders, focusing on metabotropic glutamate (mGlu) receptor biology in anxiety disorders and on anxiety-related biomarkers. OBJECTIVES This review aims to (1) examine recent progress in translational medicine, emphasizing the role that translational research has played in understanding of the potential of mGlu receptor agonists and antagonists as anxiolytics, (2) identify lacunas where animal and human research have yet to be connected, and (3) suggest areas where translational research can be further developed. RESULTS Current data show that animal and human mGlu(5) binding can be directly compared in experiments using the PET ligand (11)C-ABP688. Testing of the mGlu(2/3) receptor agonist LY354740 in the fear-potentiated startle paradigm allows direct functional comparisons between animals and humans. LY354740 has been tested in panic models, but in different models in rats and humans, hindering efforts at translation. Other potentially translatable methods, such as stress-induced hyperthermia and HPA-axis measures, either have been underexploited or are associated with technical difficulties. New techniques such as quantitative trait loci (QTL) analysis may be useful for generating novel biomarkers of anxiety. CONCLUSIONS Translational medicine approaches can be valuable to the development of anxiolytics, but the amount of cross-fertilization between clinical and pre-clinical departments will need to be expanded to realize the full potential of these approaches.
Collapse
|
11
|
Danhof M, de Jongh J, De Lange ECM, Della Pasqua O, Ploeger BA, Voskuyl RA. Mechanism-Based Pharmacokinetic-Pharmacodynamic Modeling: Biophase Distribution, Receptor Theory, and Dynamical Systems Analysis. Annu Rev Pharmacol Toxicol 2007; 47:357-400. [PMID: 17067280 DOI: 10.1146/annurev.pharmtox.47.120505.105154] [Citation(s) in RCA: 203] [Impact Index Per Article: 11.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Mechanism-based PK-PD models differ from conventional PK-PD models in that they contain specific expressions to characterize, in a quantitative manner, processes on the causal path between drug administration and effect. This includes target site distribution, target binding and activation, pharmacodynamic interactions, transduction, and homeostatic feedback mechanisms. As the final step, the effects on disease processes and disease progression are considered. Particularly through the incorporation of concepts from receptor theory and dynamical systems analysis, important progress has been made in the field of mechanism-based PK-PD modeling. This has yielded models with much-improved properties for extrapolation and prediction. These models constitute a theoretical basis for rational drug discovery and development.
Collapse
Affiliation(s)
- Meindert Danhof
- Leiden/Amsterdam Center for Drug Research, Division of Pharmacology, Leiden University, 2300 RA Leiden, The Netherlands.
| | | | | | | | | | | |
Collapse
|
12
|
Kshirsagar SA, Blaschke TF, Sheiner LB, Krygowski M, Acosta EP, Verotta D. Improving data reliability using a non-compliance detection method versus using pharmacokinetic criteria. J Pharmacokinet Pharmacodyn 2006; 34:35-55. [PMID: 17004125 DOI: 10.1007/s10928-006-9032-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2005] [Accepted: 08/18/2006] [Indexed: 12/14/2022]
Abstract
Data from clinical trials present numerous problems for the data analyst. These include non-compliance with the prescribed dosing regimen and inaccurate recollection of dosing history by patients as well as mistakes in recording data. Several methods have been proposed to address these issues. One such technique by Lu et al. (Selecting reliable pharmacokinetic data for explanatory analyses of clinical trials in the presence of possible noncompliance. J. Pharmacokinet. Pharmacodyn. 28:343-362 (2001)) identifies occasions in pharmacokinetic (PK) data where the preceding dosing history is likely to be unreliable. We used this method, implemented in the software program NONMEM (beta) VI, to clean a dataset containing indinavir (IDV) plasma concentrations from HIV-1 infected patients. The data was also cleaned by inspection in Microsoft Excel using clinical PK criteria. A one-compartment model with first order absorption and elimination was fit to both sets of cleaned data. IDV population PK parameters obtained from these analyses were similar to those reported previously. It is established that IDV nephrotoxicity is related to high IDV exposure. However, no relationships were found between any PK parameters and nephrotoxicity in the "compliance cleaned" dataset. In the "PK cleaned" dataset, the oral clearance and apparent volume were lower by 9.1% and 6.6%, respectively in patients with any type of nephrotoxicity and the maximum IDV concentration (C(max)) was 12.1% higher. In patients suffering from nephrolithiasis in particular, C(max) was 15.5% higher. Accordingly, the use of the non-compliance detection method did not improve the reliability of our dataset over the usual method of applying clinical criteria. In fact, analyses on the compliance-cleaned dataset missed some exposure-toxicity relationships. Thus, automated methods must be tested rigorously with 'real life' datasets, used with caution, and always in conjunction with clinical reasoning to avoid overlooking a signal in noisy data.
Collapse
Affiliation(s)
- Smita A Kshirsagar
- Department of Medicine, Stanford University Medical Center, Stanford, CA, USA
| | | | | | | | | | | |
Collapse
|
13
|
Zhang L, Sinha V, Forgue ST, Callies S, Ni L, Peck R, Allerheiligen SRB. Model-based drug development: the road to quantitative pharmacology. J Pharmacokinet Pharmacodyn 2006; 33:369-93. [PMID: 16770528 DOI: 10.1007/s10928-006-9010-8] [Citation(s) in RCA: 76] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2005] [Accepted: 02/14/2006] [Indexed: 10/24/2022]
Abstract
High development costs and low success rates in bringing new medicines to the market demand more efficient and effective approaches. Identified by the FDA as a valuable prognostic tool for fulfilling such a demand, model-based drug development is a mathematical and statistical approach that constructs, validates, and utilizes disease models, drug exposure-response models, and pharmacometric models to facilitate drug development. Quantitative pharmacology is a discipline that learns and confirms the key characteristics of new molecular entities in a quantitative manner, with goal of providing explicit, reproducible, and predictive evidence for optimizing drug development plans and enabling critical decision making. Model-based drug development serves as an integral part of quantitative pharmacology. This work reviews the general concept, basic elements, and evolving role of model-based drug development in quantitative pharmacology. Two case studies are presented to illustrate how the model-based drug development approach can facilitate knowledge management and decision making during drug development. The case studies also highlight the organizational learning that comes through implementation of quantitative pharmacology as a discipline. Finally, the prospects of quantitative pharmacology as an emerging discipline are discussed. Advances in this discipline will require continued collaboration between academia, industry and regulatory agencies.
Collapse
|
14
|
Chien JY, Friedrich S, Heathman MA, de Alwis DP, Sinha V. Pharmacokinetics/Pharmacodynamics and the stages of drug development: role of modeling and simulation. AAPS JOURNAL 2005; 7:E544-59. [PMID: 16353932 PMCID: PMC2751257 DOI: 10.1208/aapsj070355] [Citation(s) in RCA: 138] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
Pharmacokinetic (PK) and pharmacodynamic (PD) modeling and simulation (M&S) are well-recognized powerful tools that enable effective implementation of the learn-and-confirm paradigm in drug development. The impact of PK/PD M&S on decision making and drug development risk management is dependent on the question being asked and on the availability and quality of data accessible at a particular stage of drug development. For instance, M&S methodologies can be used to capture uncertainty and use the expected variability in PK/PD data generated in preclinical species for projection of the plausible range of clinical dose; clinical trial simulation can be used to forecast the probability of achieving a target response in patients based on information obtained in early phases of development. Framing the right question and capturing the key assumptions are critical components of the "learn-and-confirm" paradigm in the drug development process and are essential to delivering high-value PK/PD M&S results. Selected works of PK/PD modeling and simulation from preclinical to phase III are presented as case examples in this article.
Collapse
Affiliation(s)
- Jenny Y Chien
- Lilly Research Laboratories, Eli Lilly & Company, Indianapolis, IN 46285, USA.
| | | | | | | | | |
Collapse
|
15
|
|
16
|
Abstract
Pharmacodynamics is the study of the time course of pharmacological effects of drugs. The field of pharmacodynamic modeling has made many advances, due in part to the relatively recent development of basic and extended mechanism-based models. The purpose of this article is to describe the classic as well as contemporary approaches, with an emphasis on pertinent equations and salient model features. In addition, current methods of integrating various system complexities into these models are discussed. Future pharmacodynamic models will most likely reflect an assembly of the basic components outlined in this review.
Collapse
Affiliation(s)
- Donald E Mager
- Department of Pharmaceutical Sciences, School of Pharmacy and Pharmaceutical Sciences, University at Buffalo, State University of New York, Buffalo, New York 14260, USA
| | | | | |
Collapse
|
17
|
Cohen JS. Why aren't lower, effective, OTC doses available earlier by prescription? Ann Pharmacother 2003; 37:136-42. [PMID: 12503949 DOI: 10.1345/aph.1a487] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022] Open
Abstract
BACKGROUND Many popular oral over-the-counter (OTC) drugs were originally available only by prescription, but not at the low doses contained in their OTC counterparts. Yet, if OTC doses are effective for treating mild symptoms, why weren't these low, often safer doses made available at least by prescription when the drugs were first approved? OBJECTIVE To examine issues surrounding the delayed approval of OTC doses by the Food and Drug Administration (FDA). METHODS Information reviewed included package inserts, data obtained from manufacturers, and articles published in MEDLINE (1966 to December 2001). Medications examined included presently available and potentially approved OTC antiinflammatory, gastrointestinal, and antihistamine drugs. RESULTS Considerable data demonstrate the effectiveness of ibuprofen, naproxen, ranitidine, famotidine, nizatidine, diphenhydramine, and clemastine at OTC doses. Published studies also show the effectiveness of celecoxib, omeprazole, and fexofenadine at doses 33-50% lower than currently recommended for prescription use. CONCLUSIONS OTC doses are effective for many patients with mild symptoms and for some with serious symptoms. However, OTC-like doses are usually not offered when drugs are approved for prescription use because new drugs are usually studied in patients with serious conditions requiring higher doses; manufacturers and the FDA seem to prefer a middle-dose approach; >75% of subjects in premarketing dose studies are male; and averaging the responses of study subjects may obscure a wide range of interindividual variation in drug response. Simplistic dosage guidelines make therapeutic decisions easier. Because dose-related adverse effects frequently diminish quality-of-life and reduce adherence, the early availability of OTC-like doses, at least by prescription, would allow healthcare professionals greater flexibility in matching medication doses to patients' widely differing needs.
Collapse
Affiliation(s)
- Jay S Cohen
- Department of Family and Preventive Medicine, Department of Psychiatry, University of California-San Diego, La Jolla, CA, USA.
| |
Collapse
|
18
|
Abstract
Drug selection is now widely viewed as an important and relatively new, yet largely unsolved, bottleneck in the drug discovery and development process. In order to achieve an efficient selection process, high quality, rapid, predictive and correlative ADME models are required in order for them to be confidently used to support critical financial decisions. Systems that can be relied upon to accurately predict performance in humans have not existed, and decisions have been made using tools whose capabilities could not be verified until candidates went to clinical trial, leading to the high failure rates historically observed. However, with the sequencing of the human genome, advances in proteomics, the anticipation of the identification of a vastly greater number of potential targets for drug discovery, and the potential of pharmacogenomics to require individualized evaluation of drug kinetics as well as drug effects, there is an urgent need for rapid and accurately computed pharmacokinetic properties.
Collapse
Affiliation(s)
- George M Grass
- LION bioscience, 9880 Campus Point Drive, San Diego, CA 92121, USA
| | | |
Collapse
|
19
|
Meibohm B, Derendorf H. Pharmacokinetic/pharmacodynamic studies in drug product development. J Pharm Sci 2002; 91:18-31. [PMID: 11782894 DOI: 10.1002/jps.1167] [Citation(s) in RCA: 111] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
In the quest of ways for rationalizing and accelerating drug product development, integrated pharmacokinetic/pharmacodynamic (PK/PD) concepts provide a highly promising tool. PK/PD modeling concepts can be applied in all stages of preclinical and clinical drug development, and their benefits are multifold. At the preclinical stage, potential applications might comprise the evaluation of in vivo potency and intrinsic activity, the identification of bio-/surrogate markers, as well as dosage form and regimen selection and optimization. At the clinical stage, analytical PK/PD applications include characterization of the dose-concentration-effect/toxicity relationship, evaluation of food, age and gender effects, drug/drug and drug/disease interactions, tolerance development, and inter- and intraindividual variability in response. Predictive PK/PD applications can also involve extrapolation from preclinical data, simulation of drug responses, as well as clinical trial forecasting. Rigorous implementation of the PK/PD concepts in drug product development provides a rationale, scientifically based framework for efficient decision making regarding the selection of potential drug candidates, for maximum information gain from the performed experiments and studies, and for conducting fewer, more focused clinical trials with improved efficiency and cost effectiveness. Thus, PK/PD concepts are believed to play a pivotal role in streamlining the drug development process of the future.
Collapse
Affiliation(s)
- Bernd Meibohm
- Department of Pharmaceutical Sciences, College of Pharmacy, University of Tennessee, 874 Union Avenue, Room 5p, Memphis, Tennessee 38163, USA.
| | | |
Collapse
|
20
|
Grass GM, Sinko PJ. Effect of diverse datasets on the predictive capability of ADME models in drug discovery. Drug Discov Today 2001. [DOI: 10.1016/s1359-6446(01)00150-7] [Citation(s) in RCA: 21] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
21
|
Gomeni R, Bani M, D'Angeli C, Corsi M, Bye A. Computer-assisted drug development (CADD): an emerging technology for designing first-time-in-man and proof-of-concept studies from preclinical experiments. Eur J Pharm Sci 2001; 13:261-70. [PMID: 11384848 DOI: 10.1016/s0928-0987(01)00111-7] [Citation(s) in RCA: 21] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
Abstract
Computer-assisted drug development (CADD) is an emerging technology for accelerating drug development based on the integration of mathematical modelling and simulation. This methodology provides a knowledge-based decisional tool on alternative development strategies based on the evaluation of potential risks on drug safety, and the definition of experimental design of new trials with expected power and probability of success. An example of CADD implementation is presented to design the first-time-in-man (FTIM) and the proof-of-concept (PoC) study of a new CNS compound. The final objective of the example presented is not necessarily to supply a success story of a correct prediction of human data from animal studies but to define a credible strategy suitable to design FTIM and PoC studies using preclinical data without the support of any human in vivo information. Rhesus monkey and human PK were initially estimated using allometric scaling on data collected in dogs, cynomolgus monkeys and rats. A PK/PD model was derived from a study conducted in rodent and validated by comparing the model predicted response to the one observed in a PET experiment conducted in rhesus monkey. The final PK/PD model, incorporating potential variability and uncertainty on scaled human prediction together with a receptor affinity adjustment derived from in vitro binding studies, was used to design the first-time-in-man and the proof-of-concept study.
Collapse
Affiliation(s)
- R Gomeni
- Department of Experimental Medicine, GlaxoSmithKline Group, GlaxoWellcome S.P.A., Via A. Fleming 2, 37135 Verona, Italy.
| | | | | | | | | |
Collapse
|
22
|
Derendorf H, Lesko LJ, Chaikin P, Colburn WA, Lee P, Miller R, Powell R, Rhodes G, Stanski D, Venitz J. Pharmacokinetic/Pharmacodynamic Modeling in Drug Research and Development. J Clin Pharmacol 2000. [DOI: 10.1177/009127000004001211] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Affiliation(s)
| | - Lawrence J. Lesko
- Center for Drug Evaluation and Research, Food and Drug Administration (FDA), Rockville, Maryland
| | | | | | - Peter Lee
- Center for Drug Evaluation and Research, Food and Drug Administration (FDA), Rockville, Maryland
| | | | | | | | | | | |
Collapse
|
23
|
Abstract
We propose a framework for considering the role of pharmacokinetic/pharmacodynamic modeling in drug development and an appraisal of its current and potential impact on that activity. After some introduction, definitions, and background information on drug development, we discuss subject-matter models that underlie pharmacokinetic/pharmacodynamic modeling and show how they determine appropriate statistical models. We discuss the broad role modeling can play in drug development, enhancing primarily the "learning" steps, i.e. acquiring the information needed for the label and for planning efficient confirmatory clinical trials. Examples of past applications of modeling to drug development are presented in tabular form, followed by a discussion of some practical issues in application. Modeling will not reach its potential utility until it is manifest as a visible and separate work unit within a drug development program. We suggest that that work unit is the "in numero" study: a protocol-driven exercise designed to extract additional information, and/or answer a specific drug-development question, through an integrated model-based (meta-) analysis of existent raw data, often pooled across separate (clinical) studies.
Collapse
Affiliation(s)
- L B Sheiner
- Department of Laboratory Medicine, University of California San Francisco 94143, USA.
| | | |
Collapse
|
24
|
Abstract
No Abstract
Collapse
Affiliation(s)
- Alain Patat
- Wyeth Ayerst Research, Clinical Pharmacology, 80 avenue de Général de Gaulle, 92031 Paris La Défense, France
| |
Collapse
|
25
|
Abstract
Sildenafil is highly effective for treating erectile dysfunction (ED). However, its use has been associated with serious adverse events including myocardial infarctions and strokes, and 130 verifiable plus 112 unverified deaths reported to the US Food and Drug Administration during the 8 months after sildenafil was introduced in the US, and 522 reported deaths during the 13.5 months after its introduction. Moreover, some events have occurred in men taking their first dose of the agent, suggesting that sildenafil, like some drugs that affect blood pressure, may provoke a first-dose reaction. This possibility warrants extra caution to be used when initiating treatment with sildenafil. Such caution is not currently provided by the current dosage guidelines that, for example, recommend the use of sildenafil 50 mg initially for most men between the ages of 18 and 65 years, despite wide differences in bodyweight, age, drug metabolism, health status and usage of other medications. It can be difficult to identify the patient who may be unusually sensitive to the effects of sildenafil. Exercise stress tests have been recommended, but serious adverse events have occurred in men with normal stress tests following the ingestion of sildenafil. Blood pressure monitoring following sildenafil administration will not prevent a serious adverse drug event already in progress. This article discusses the advantages and disadvantages of initiating treatment with a low test dose of sildenafil, performed at home or in the doctor's office. The advantages of this approach include: (i) identifying patients who are highly sensitive to the effects of sildenafil and who may need no higher dose; (ii) minimising adverse effects such as flushing and dizziness that often frighten patients and may affect adherence; (iii) avoidance of major adverse events; and (iv) reassuring patients with ED who remain wary about trying sildenafil therapy.
Collapse
Affiliation(s)
- J S Cohen
- Department of Psychiatry, University of California, San Diego, La Jolla 92093, USA.
| |
Collapse
|
26
|
Yamaguchi J, Watanabe Y, Ohmichi M, Jingu S, Ogawa N, Kokatsu J, Fukushima K, Goto J. Ultrasensitive determination of NE-100, a novel sigma ligand, in human plasma by liquid chromatography and electrospray ionization tandem mass spectrometry combined with a column-switching technique. JOURNAL OF CHROMATOGRAPHY. B, BIOMEDICAL SCIENCES AND APPLICATIONS 1999; 730:61-70. [PMID: 10437673 DOI: 10.1016/s0378-4347(99)00183-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
For the highly sensitive and selective determination of NE-100, a novel sigma ligand, at levels of low picogram per milliliter of human plasma, a method with excellent reliability employing liquid chromatography (LC)-electrospray ionization (ESI) tandem mass spectrometry (MS-MS) combined with a column-switching technique has been developed. The method involves the use of a stable isotope labeled compound as the internal standard (I.S.), liquid-solid extraction of a plasma specimen with a C8 cartridge, automated on-line clean-up on a short trapping column, subsequent separation on a micro-bore C18 column and detection with ESI-MS-MS using m/z 356 ([M+H]+) as a precursor ion and m/z 105 as a product ion in a selected reaction monitoring mode. The detection and the quantification limits of NE-100 in plasma were 0.5 pg/ml with a signal-to-noise ratio (S/N) of 3 and 2.3 pg/ml, respectively, with an S/N of 21. The good linearity of the calibration graph was obtained in the range of 2.3 to approximately 907.0 pg/ml with excellent reliability. The developed method was applied to the determination of NE-100 in plasma obtained from the clinical trail.
Collapse
Affiliation(s)
- J Yamaguchi
- Pharmaceutical Research Laboratories, Taisho Pharmaceutical Co., Ltd., Ohmiya-shi, Saitama, Japan
| | | | | | | | | | | | | | | |
Collapse
|
27
|
Derendorf H, Meibohm B. Modeling of pharmacokinetic/pharmacodynamic (PK/PD) relationships: concepts and perspectives. Pharm Res 1999; 16:176-85. [PMID: 10100300 DOI: 10.1023/a:1011907920641] [Citation(s) in RCA: 245] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
Pharmacokinetic/pharmacodynamic (PK/PD)-modeling links dose-concentration relationships (PK) and concentration-effect relationships (PD), thereby facilitating the description and prediction of the time course of drug effects resulting from a certain dosing regimen. PK/PD-modeling approaches can basically be distinguished by four major attributes. The first characterizes the link between measured drug concentration and the response system, direct link versus indirect link. The second considers how the response system relates effect site concentration to the observed outcome, direct versus indirect response. The third regards what clinically or experimentally assessed information is used to establish the link between concentration and effect, hard link versus soft link. And the fourth considers the time dependency of pharmacodynamic model parameters, distinguishing between time-variant versus time-invariant. Application of PK/PD-modeling concepts has been identified as potentially beneficial in all phases of preclinical and clinical drug development. Although today predominantly limited to research, broader application of PK/PD-concepts in clinical therapy will provide a more rational basis for patient-specific dosage individualization and may thus guide applied pharmacotherapy to a higher level of performance.
Collapse
Affiliation(s)
- H Derendorf
- Department of Pharmaceutics, College of Pharmacy, University of Florida, Gainesville 32610, USA.
| | | |
Collapse
|
28
|
Abstract
Population pharmacokinetics or pharmacodynamics is the study of the variability in drug concentration or pharmacological effect between individuals when standard dosage regimens are administered. We provide an overview of pharmacokinetic models, pharmacodynamic models, population models and residual error models. We outline how population modelling approaches seek to explain interpatient variability with covariate analysis, and, in some approaches, to characterize the unexplained interindividual variability. The interpretation of the results of population modelling approaches is facilitated by shifting the emphasis from the perspective of the modeller to the perspective of the clinician. Both the explained and unexplained interpatient variability should be presented in terms of their impact on the dose-response relationship. Clinically relevant questions relating to the explained and unexplained variability in the population can be posed to the model, and confidence intervals can be obtained for the fraction of the population that is estimated to fall within a specific therapeutic range given a certain dosing regimen. Such forecasting can be used to develop optimal initial dosing guidelines. The development of population models (with random effects) permits the application of Bayes's formula to obtain improved estimates of an individual's pharmacokinetic and pharmacodynamic parameters in the light of observed responses. An important challenge to clinical pharmacology is to identify the drugs that might benefit from such adaptive-control-with-feedback dosing strategies. Drugs used for life threatening diseases with a proven pharmacokinetic-pharmacodynamic relationship, a small therapeutic range, large interindividual variability, small interoccasion variability and severe adverse effects are likely to be good candidates. Rapidly evolving changes in health care economics and consumer expectations make it unlikely that traditional drug development approaches will succeed in the future. A shift away from the narrow focus on rejecting the null hypothesis towards a broader focus on seeking to understand the factors that influence the dose-response relationship--together with the development of the next generation of software based on population models--should permit a more efficient and rational drug development programme.
Collapse
Affiliation(s)
- C Minto
- Royal North Shore Hospital, University of Sydney, Australia
| | | |
Collapse
|
29
|
Abstract
While pharmacokinetic/pharmacodynamic relationships for opioids have not been consistently demonstrable or sufficiently predictive, there remain compelling reasons to pursue such relationships. Among the reasons for pursuing pharmacokinetic/ pharmacodynamic relationships is the prospect of predicting the time-action characteristics of new therapeutics on the basis of early studies in normals using pharmacodynamic surrogates for analgesia. The realization of such a model could improve the efficiency of development of analgesics. Four studies involving 98 normals were conducted to determine whether significant and reproducible relationships existed for oxycodone in the form of an oral controlled-release tablet. All studies were analytically blinded and utilized a validated gas chromatographic/mass spectrometric, sensitive (0.2 ng/ml), and specific method for oxycodone (four studies) and oxymorphone (two studies) quantitation in 17 to 20 serial plasma samples over 36 to 48 hours following a single 20 mg (or 40 mg) dose in each study. Concurrent assessments included vital signs and opioid effect VAS questionnaires. The studies demonstrated significant relationships between plasma oxycodone (but not oxymorphone) and pharmacodynamic surrogates (particularly VAS "drug effect") and were predictive of the 12-hour duration of pain control and prompt onset of analgesia subsequently demonstrated in multiple clinical studies involving patients with various pathological pain syndromes. The results suggest that investigators can make earlier, accurate predictions of opioid analgesic pharmacodynamics in patients based on pharmacokinetic/pharmacodynamic studies in normal volunteers.
Collapse
Affiliation(s)
- R F Kaiko
- Purdue Frederick Company, Norwalk, Connecticut, USA
| |
Collapse
|
30
|
Jerling M. Population pharmacokinetics and pharmacodynamics: potential use for gathering dose-concentration-response. Eur J Drug Metab Pharmacokinet 1996; 21:113-21. [PMID: 8839684 DOI: 10.1007/bf03190259] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
The population approach is a general term covering different aspects of kinetic and dynamic data collected mainly from drug-treated patients and new techniques allowing evaluation of sparse observations from each subject. Such data originate from clinically relevant conditions and can give information on several qualities of a drug. An example is given with the tricyclic antidepressant nortriptyline for which the kinetics and the concentration-effect relationship have been thoroughly documented previously with conventional techniques. We have evaluated retrospective data from a therapeutic drug monitoring service using a nonparametric population kinetic method (NPML) that allows description of kinetic outliers and nonlinear relationships between kinetic parameters and covariates. In addition, drug interactions, nonlinear kinetics and dosing habits were studied with other techniques corroborating previous results and adding new information. The concentration-effect relationship could not be evaluated from our data as information on efficacy and adverse effects was of too low quality. However, several controlled studies have defined a therapeutic concentration interval and a discussion on dosing strategies is based on this interval. Collection of sparse data in patients during phases II-IV of drug development as a complement to conventional studies is highly recommendable.
Collapse
Affiliation(s)
- M Jerling
- Department of Clinical Pharmacology, Karolinska Institute, Huddinge University Hospital, Sweden
| |
Collapse
|
31
|
Braggio S, Barnaby RJ, Grossi P, Cugola M. A strategy for validation of bioanalytical methods. J Pharm Biomed Anal 1996; 14:375-88. [PMID: 8729635 DOI: 10.1016/0731-7085(95)01644-9] [Citation(s) in RCA: 139] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Affiliation(s)
- S Braggio
- Drug Metabolism Department, Glaxo-Wellcome Research Laboratories, Verona, Ita
| | | | | | | |
Collapse
|