Estimating the unknown parameters of the natural history of metachronous colorectal cancer using discrete-event simulation

A Review on Calibration Methods of Cancer Simulation Models

Calibration, a critical step in the development of simulation models, involves adjusting unobservable parameters to ensure that the outcomes of the model closely align with observed target data. This process is particularly vital in cancer simulation models with a natural history component where direct data to inform natural history parameters are rarely available. This work reviews the literature of cancer simulation models with a natural history component and identifies the calibration approaches used in these models with respect to the following attributes: calibration target, goodness-of-fit (GOF) measure, parameter search algorithm, acceptance criteria, and stopping rules. After a comprehensive search of the PubMed database from 1981 to June 2023, 68 studies were included in the review. Nearly all (n=66) articles specified the calibration targets, and most articles (n=56) specified the parameter search algorithms they used, whereas goodness-of-fit metric (n=51) and acceptance criteria/stopping rule (n=45) were reported for fewer times. The most frequently used calibration targets were incidence, mortality, and prevalence, whose data sources primarily come from cancer registries and observational studies. The most used goodness-of-fit measure was weighted mean squared error. Random search has been the predominant method for parameter search, followed by grid search and Nelder-mead method. Machine learning-based algorithms, despite their fast advancement in the recent decade, has been underutilized in the cancer simulation models. More research is needed to compare different parameter search algorithms used for calibration.

Key points

This work reviewed the literature of cancer simulation models with a natural history component and identified the calibration approaches used in these models with respect to the following attributes: calibration target, goodness-of-fit (GOF) measure, parameter search algorithm, acceptance criteria, and stopping rules.Random search has been the predominant method for parameter search, followed by grid search and Nelder-mead method.Machine learning-based algorithms, despite their fast advancement in the recent decade, has been underutilized in the cancer simulation models. Furthermore, more research is needed to compare different parameter search algorithms used for calibration.

A two-phase approach to re-calibrating expensive computer simulation for sex-specific colorectal neoplasia development modeling

BACKGROUND

Medical evidence from more recent observational studies may significantly alter our understanding of disease incidence and progression, and would require recalibration of existing computational and predictive disease models. However, it is often challenging to perform recalibration when there are a large number of model parameters to be estimated. Moreover, comparing the fitting performances of candidate parameter designs can be difficult due to significant variation in simulated outcomes under limited computational budget and long runtime, even for one simulation replication.

METHODS

We developed a two-phase recalibration procedure. As a proof-of-the-concept study, we verified the procedure in the context of sex-specific colorectal neoplasia development. We considered two individual-based state-transition stochastic simulation models, estimating model parameters that govern colorectal adenoma occurrence and its growth through three preclinical states: non-advanced precancerous polyp, advanced precancerous polyp, and cancerous polyp. For the calibration, we used a weighted-sum-squared error between three prevalence values reported in the literature and the corresponding simulation outcomes. In phase 1 of the calibration procedure, we first extracted the baseline parameter design from relevant studies on the same model. We then performed sampling-based searches within a proper range around the baseline design to identify the initial set of good candidate designs. In phase 2, we performed local search (e.g., the Nelder-Mead algorithm), starting from the candidate designs identified at the end of phase 1. Further, we investigated the efficiency of exploring dimensions of the parameter space sequentially based on our prior knowledge of the system dynamics.

RESULTS

The efficiency of our two-phase re-calibration procedure was first investigated with CMOST, a relatively inexpensive computational model. It was then further verified with the V/NCS model, which is much more expensive. Overall, our two-phase procedure showed a better goodness-of-fit than the straightforward employment of the Nelder-Mead algorithm, when only a limited number of simulation replications were allowed. In addition, in phase 2, performing local search along parameter space dimensions sequentially was more efficient than performing the search over all dimensions concurrently.

CONCLUSION

The proposed two-phase re-calibration procedure is efficient at estimating parameters of computationally expensive stochastic dynamic disease models.

Synchronous Neoplastic Lesions In Referred Patients With Colorectal Cancer: A Retrospective Cohort Study

Background

Synchronous neoplastic lesions are usually present in patients with colorectal cancer (CRC) at diagnosis or postoperative follow-up endoscopy. However, few studies have been published about the clinicopathological features of synchronous lesions, especially those of synchronous advanced neoplasia. This study aimed to describe synchronous lesions in patients with CRC because this knowledge may be useful for preventing the development of metachronous cancer.

Material and methods

We retrospectively reviewed 261 primary CRC cases with synchronous lesions referred to our hospital during a 4-year period. Personal history, habits, family history, characteristics of index cancer, and synchronous lesions were assessed.

Results

In total, the 261 patients with CRC had 812 synchronous adenomas and 146 advanced neoplasia. Diminutive, small, and large polyps made up 66.7%, 20.2%, and 13.1% of all lesions, respectively; 9.3% of diminutive and small adenomas were advanced neoplasia, and 45.2% of synchronous advanced lesions were subcentimeter polyps. Both synchronous non-advanced lesions and advanced lesions developed most frequently in the distal colon, followed by the proximal colon, and were least frequently found in the rectum (P < 0.001). Older age (P = 0.04) and male gender (P = 0.001) were associated with the presence of advanced neoplasia in CRC cases with synchronous neoplastic lesions. Meanwhile, the use of aspirin may be associated with a lower incidence of advanced neoplasia (P = 0.04).

Conclusion

Patients diagnosed with CRC require detailed clearing of the remainder of the colon at baseline coloscopy or postoperative follow-up examination, and we should take a more cautious approach to synchronous subcentimeter polyps in this group of patients.

Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

Background. Developing efficient procedures of model calibration, which entails matching model predictions to observed outcomes, has gained increasing attention. With faithful but complex simulation models established for cancer diseases, key parameters of cancer natural history can be investigated for possible fits, which can subsequently inform optimal prevention and treatment strategies. When multiple calibration targets exist, one approach to identifying optimal parameters relies on the Pareto frontier. However, computational burdens associated with higher-dimensional parameter spaces require a metamodeling approach. The goal of this work is to explore multiobjective calibration using Gaussian process regression (GPR) with an eye toward how multiple goodness-of-fit (GOF) criteria identify Pareto-optimal parameters. Methods. We applied GPR, a metamodeling technique, to estimate colorectal cancer (CRC)-related prevalence rates simulated from a microsimulation model of CRC natural history, known as the Colon Modeling Open Source Tool (CMOST). We embedded GPR metamodels within a Pareto optimization framework to identify best-fitting parameters for age-, adenoma-, and adenoma staging-dependent transition probabilities and risk factors. The Pareto frontier approach is demonstrated using genetic algorithms with both sum-of-squared errors (SSEs) and Poisson deviance GOF criteria. Results. The GPR metamodel is able to approximate CMOST outputs accurately on 2 separate parameter sets. Both GOF criteria are able to identify different best-fitting parameter sets on the Pareto frontier. The SSE criterion emphasizes the importance of age-specific adenoma progression parameters, while the Poisson criterion prioritizes adenoma-specific progression parameters. Conclusion. Different GOF criteria assert different components of the CRC natural history. The combination of multiobjective optimization and nonparametric regression, along with diverse GOF criteria, can advance the calibration process by identifying optimal regions of the underlying parameter landscape.

Nonidentifiability in Model Calibration and Implications for Medical Decision Making

BACKGROUND

Calibration is the process of estimating parameters of a mathematical model by matching model outputs to calibration targets. In the presence of nonidentifiability, multiple parameter sets solve the calibration problem, which may have important implications for decision making. We evaluate the implications of nonidentifiability on the optimal strategy and provide methods to check for nonidentifiability.

METHODS

We illustrate nonidentifiability by calibrating a 3-state Markov model of cancer relative survival (RS). We performed 2 different calibration exercises: 1) only including RS as a calibration target and 2) adding the ratio between the 2 nondeath states over time as an additional target. We used the Nelder-Mead (NM) algorithm to identify parameter sets that best matched the calibration targets. We used collinearity and likelihood profile analyses to check for nonidentifiability. We then estimated the benefit of a hypothetical treatment in terms of life expectancy gains using different, but equally good-fitting, parameter sets. We also applied collinearity analysis to a realistic model of the natural history of colorectal cancer.

RESULTS

When only RS is used as the calibration target, 2 different parameter sets yield similar maximum likelihood values. The high collinearity index and the bimodal likelihood profile on both parameters demonstrated the presence of nonidentifiability. These different, equally good-fitting parameter sets produce different estimates of the treatment effectiveness (0.67 v. 0.31 years), which could influence the optimal decision. By incorporating the additional target, the model becomes identifiable with a collinearity index of 3.5 and a unimodal likelihood profile.

CONCLUSIONS

In the presence of nonidentifiability, equally likely parameter estimates might yield different conclusions. Checking for the existence of nonidentifiability and its implications should be incorporated into standard model calibration procedures.

Risk factors for metachronous colorectal cancer following a primary colorectal cancer: A prospective cohort study

Individuals diagnosed with colorectal cancer (CRC) are at risk of developing a metachronous CRC. We examined the associations between personal, tumour-related and lifestyle risk factors, and risk of metachronous CRC. A total of 7,863 participants with incident colon or rectal cancer who were recruited in the USA, Canada and Australia to the Colon Cancer Family Registry during 1997-2012, except those identified as high-risk, for example, Lynch syndrome, were followed up approximately every 5 years. We estimated the risk of metachronous CRC, defined as the first new primary CRC following an interval of at least one year after the initial CRC diagnosis. Observation time started at the age at diagnosis of the initial CRC and ended at the age at diagnosis of the metachronous CRC, last contact or death whichever occurred earliest, or were censored at the age at diagnosis of any metachronous colorectal adenoma. Cox regression was used to derive hazard ratios (HRs) and 95% confidence intervals (CIs). During a mean follow-up of 6.6 years, 142 (1.81%) metachronous CRCs were diagnosed (mean age at diagnosis 59.8; incidence 2.7/1,000 person-years). An increased risk of metachronous CRC was associated with the presence of a synchronous CRC (HR = 2.73; 95% CI: 1.30-5.72) and the location of cancer in the proximal colon at initial diagnosis (compared with distal colon or rectum, HR = 4.16; 95% CI: 2.80-6.18). The presence of a synchronous CRC and the location of the initial CRC might be useful for deciding the intensity of surveillance colonoscopy for individuals diagnosed with CRC.

Cost-effectiveness of alternative colonoscopy surveillance strategies to mitigate metachronous colorectal cancer incidence

BACKGROUND

The incidence of metachronous colorectal cancer (MCRC) among colorectal cancer (CRC) survivors varies significantly, and the optimal colonoscopy surveillance practice for mitigating MCRC incidence is unknown.

METHODS

A cost-effectiveness analysis was used to compare the performances of the US Multi-Society Task Force guideline and all clinically reasonable colonoscopy surveillance strategies for 50- to 79-year-old posttreatment CRC patients with a computer simulation model.

RESULTS

The US guideline [(1,3,5)] recommends the first colonoscopy 1 year after treatment, whereas the second and third colonoscopies are to be repeated at 3- and 5-year intervals. Some promising alternative cost-effective strategies were identified. In comparison with the US guideline, under various scenarios for a 20-year period, 1) reducing the surveillance interval of the guideline after the first colonoscopy by 1 year [(1,2,5)] would save up to 78 discounted life-years (LYs) and prevent 23 MCRCs per 1000 patients (incremental cost-effectiveness ratio [ICER] ≤ $23,270/LY), 2) reducing the intervals after the first and second negative colonoscopies by 1 year [(1,2,4)] would save/prevent up to 109 discounted LYs and 36 MCRCs (ICER ≤ $52,155/LY), and 3) reducing the surveillance intervals after the first and second negative colonoscopy by 1 and 2 years [(1,2,3)] would save/prevent up to 141 discounted LYs and 50 MCRCs (ICER ≤ $63,822/LY). These strategies would require up to 1100 additional colonoscopies per 1000 patients. Although the US guideline might not be cost-effective in comparison with a less intensive oncology guideline [(3,3,5); the ICER could be as high as $140,000/LY], the promising strategies would be cost-effective in comparison with such less intensive guidelines unless the cumulative MCRC incidence were very low.

CONCLUSIONS

The US guideline might be improved by a slight increase in the surveillance intensity at the expense of moderately increased cost. More research is warranted to explore the benefits/harms of such practices. Cancer 2016;122:2560-70. © 2016 American Cancer Society.

A method for using real world data in breast cancer modeling

OBJECTIVES

Today, hospitals and other health care-related institutions are accumulating a growing bulk of real world clinical data. Such data offer new possibilities for the generation of disease models for the health economic evaluation. In this article, we propose a new approach to leverage cancer registry data for the development of Markov models. Records of breast cancer patients from a clinical cancer registry were used to construct a real world data driven disease model.

METHODS

We describe a model generation process which maps database structures to disease state definitions based on medical expert knowledge. Software was programmed in Java to automatically derive a model structure and transition probabilities. We illustrate our method with the reconstruction of a published breast cancer reference model derived primarily from clinical study data. In doing so, we exported longitudinal patient data from a clinical cancer registry covering eight years. The patient cohort (n=892) comprised HER2-positive and HER2-negative women treated with or without Trastuzumab.

RESULTS

The models generated with this method for the respective patient cohorts were comparable to the reference model in their structure and treatment effects. However, our computed disease models reflect a more detailed picture of the transition probabilities, especially for disease free survival and recurrence.

CONCLUSIONS

Our work presents an approach to extract Markov models semi-automatically using real world data from a clinical cancer registry. Health care decision makers may benefit from more realistic disease models to improve health care-related planning and actions based on their own data.

Using Active Learning for Speeding up Calibration in Simulation Models

BACKGROUND

Most cancer simulation models include unobservable parameters that determine disease onset and tumor growth. These parameters play an important role in matching key outcomes such as cancer incidence and mortality, and their values are typically estimated via a lengthy calibration procedure, which involves evaluating a large number of combinations of parameter values via simulation. The objective of this study is to demonstrate how machine learning approaches can be used to accelerate the calibration process by reducing the number of parameter combinations that are actually evaluated.

METHODS

Active learning is a popular machine learning method that enables a learning algorithm such as artificial neural networks to interactively choose which parameter combinations to evaluate. We developed an active learning algorithm to expedite the calibration process. Our algorithm determines the parameter combinations that are more likely to produce desired outputs and therefore reduces the number of simulation runs performed during calibration. We demonstrate our method using the previously developed University of Wisconsin breast cancer simulation model (UWBCS).

RESULTS

In a recent study, calibration of the UWBCS required the evaluation of 378 000 input parameter combinations to build a race-specific model, and only 69 of these combinations produced results that closely matched observed data. By using the active learning algorithm in conjunction with standard calibration methods, we identify all 69 parameter combinations by evaluating only 5620 of the 378 000 combinations.

CONCLUSION

Machine learning methods hold potential in guiding model developers in the selection of more promising parameter combinations and hence speeding up the calibration process. Applying our machine learning algorithm to one model shows that evaluating only 1.49% of all parameter combinations would be sufficient for the calibration.

Do the benefits outweigh the side effects of colorectal cancer surveillance? A systematic review

Most patients treated with curative intent for colorectal cancer (CRC) are included in a follow-up program involving periodic evaluations. The survival benefits of a follow-up program are well delineated, and previous meta-analyses have suggested an overall survival improvement of 5%-10% by intensive follow-up. However, in a recent randomized trial, there was no survival benefit when a minimal vs an intensive follow-up program was compared. Less is known about the potential side effects of follow-up. Well-known side effects of preventive programs are those of somatic complications caused by testing, negative psychological consequences of follow-up itself, and the downstream impact of false positive or false negative tests. Accordingly, the potential survival benefits of CRC follow-up must be weighed against these potential negatives. The present review compares the benefits and side effects of CRC follow-up, and we propose future areas for research.

A simulation model of colorectal cancer surveillance and recurrence

Background

Approximately one-third of those treated curatively for colorectal cancer (CRC) will experience recurrence. No evidence-based consensus exists on how best to follow patients after initial treatment to detect asymptomatic recurrence. Here, a new approach for simulating surveillance and recurrence among CRC survivors is outlined, and development and calibration of a simple model applying this approach is described. The model’s ability to predict outcomes for a group of patients under a specified surveillance strategy is validated.

Methods

We developed an individual-based simulation model consisting of two interacting submodels: a continuous-time disease-progression submodel overlain by a discrete-time Markov submodel of surveillance and re-treatment. In the former, some patients develops recurrent disease which probabilistically progresses from detectability to unresectability, and which may produce early symptoms leading to detection independent of surveillance testing. In the latter submodel, patients undergo user-specified surveillance testing regimens. Parameters describing disease progression were preliminarily estimated through calibration to match five-year disease-free survival, overall survival at years 1–5, and proportion of recurring patients undergoing curative salvage surgery from one arm of a published randomized trial. The calibrated model was validated by examining its ability to predict these same outcomes for patients in a different arm of the same trial undergoing less aggressive surveillance.

Results

Calibrated parameter values were consistent with generally observed recurrence patterns. Sensitivity analysis suggested probability of curative salvage surgery was most influenced by sensitivity of carcinoembryonic antigen assay and of clinical interview/examination (i.e. scheduled provider visits). In validation, the model accurately predicted overall survival (59% predicted, 58% observed) and five-year disease-free survival (55% predicted, 53% observed), but was less accurate in predicting curative salvage surgery (10% predicted; 6% observed).

Conclusions

Initial validation suggests the feasibility of this approach to modeling alternative surveillance regimens among CRC survivors. Further calibration to individual-level patient data could yield a model useful for predicting outcomes of specific surveillance strategies for risk-based subgroups or for individuals. This approach could be applied toward developing novel, tailored strategies for further clinical study. It has the potential to produce insights which will promote more effective surveillance—leading to higher cure rates for recurrent CRC.

Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases

Longitudinal healthcare claims databases are frequently used for studying the comparative safety and effectiveness of medications, but results from these studies may be biased due to residual confounding. It is unclear whether methods for confounding adjustment that have been shown to perform well in small, simple nonrandomized studies are applicable to the large, complex pharmacoepidemiologic studies created from secondary healthcare data. Ordinary simulation approaches for evaluating the performance of statistical methods do not capture important features of healthcare claims. A statistical framework for creating replicated simulation datasets from an empirical cohort study in electronic healthcare claims data is developed and validated. The approach relies on resampling from the observed covariate and exposure data without modification in all simulated datasets to preserve the associations among these variables. Repeated outcomes are simulated using a true treatment effect of the investigator's choice and the baseline hazard function estimated from the empirical data. As an example, this framework is applied to a study of high versus low-intensity statin use and cardiovascular outcomes. Simulated data is based on real data drawn from Medicare Parts A and B linked with a prescription drug insurance claims database maintained by Caremark. Properties of the data simulated using this framework are compared with the empirical data on which the simulations were based. In addition, the simulated datasets are used to compare variable selection strategies for confounder adjustmentvia the propensity score, including high-dimensional approaches that could not be evaluated with ordinary simulation methods. The simulated datasets are found to closely resemble the observed complex data structure but have the advantage of an investigator-specified exposure effect.

Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases

Longitudinal healthcare claims databases are frequently used for studying the comparative safety and effectiveness of medications, but results from these studies may be biased due to residual confounding. It is unclear whether methods for confounding adjustment that have been shown to perform well in small, simple nonrandomized studies are applicable to the large, complex pharmacoepidemiologic studies created from secondary healthcare data. Ordinary simulation approaches for evaluating the performance of statistical methods do not capture important features of healthcare claims. A statistical framework for creating replicated simulation datasets from an empirical cohort study in electronic healthcare claims data is developed and validated. The approach relies on resampling from the observed covariate and exposure data without modification in all simulated datasets to preserve the associations among these variables. Repeated outcomes are simulated using a true treatment effect of the investigator's choice and the baseline hazard function estimated from the empirical data. As an example, this framework is applied to a study of high versus low-intensity statin use and cardiovascular outcomes. Simulated data is based on real data drawn from Medicare Parts A and B linked with a prescription drug insurance claims database maintained by Caremark. Properties of the data simulated using this framework are compared with the empirical data on which the simulations were based. In addition, the simulated datasets are used to compare variable selection strategies for confounder adjustmentvia the propensity score, including high-dimensional approaches that could not be evaluated with ordinary simulation methods. The simulated datasets are found to closely resemble the observed complex data structure but have the advantage of an investigator-specified exposure effect.

Colorectal cancer surveillance: what's new and what's next

The accumulated evidence from two decades of randomized controlled trials has not yet resolved the question of how best to monitor colorectal cancer (CRC) survivors for early detection of recurrent and metachronous disease or even whether doing so has its intended effect. A new wave of trial data in the coming years and an evolving knowledge of relevant biomarkers may bring us closer to understanding what surveillance strategies are most effective for a given subset of patients. To best apply these insights, a number of important research questions need to be addressed, and new decision making tools must be developed. In this review, we summarize available randomized controlled trial evidence comparing alternative surveillance testing strategies, describe ongoing trials in the area, and compare professional society recommendations for surveillance. In addition, we discuss innovations relevant to CRC surveillance and outline a research agenda which will inform a more risk-stratified and personalized approach to follow-up.

Cost-effectiveness of adjuvant FOLFOX and 5FU/LV chemotherapy for patients with stage II colon cancer

PURPOSE

We evaluated the cost-effectiveness of adjuvant chemotherapy using 5-fluorouracil, leucovorin (5FU/LV), and oxaliplatin (FOLFOX) compared with 5FU/LV alone and 5FU/LV compared with observation alone for patients who had resected stage II colon cancer.

METHODS

We developed 2 Markov models to represent the adjuvant chemotherapy and follow-up periods and a single Markov model to represent the observation group. We used calibration to estimate the transition probabilities among different toxicity levels. The base case considered 60-year-old patients who had undergone an uncomplicated hemicolectomy for stage II colon cancer and were medically fit to receive 6 months of adjuvant chemotherapy. We measured health outcomes in quality-adjusted life-years (QALYs) and estimated costs using 2007 US dollars.

RESULTS

In the base case, adjuvant chemotherapy of the FOLFOX regimen had an incremental cost-effectiveness ratio (ICER) of $54,359/QALY compared with the 5FU/LV regimen, and the 5FU/LV regimen had an ICER of $14,584/QALY compared with the observation group from the third-party payer perspective. The ICER values were most sensitive to 5-year relapse probability, cost of adjuvant chemotherapy, and the discount rate for the FOLFOX arm, whereas the ICER value of 5FU/LV was most sensitive to the 5-year relapse probability, 5-year survival probability, and the relapse cost. The probabilistic sensitivity analysis indicates that the ICER of 5FU/LV is less than $50,000/QALY with a probability of 99.62%, and the ICER of FOLFOX as compared with 5FU/LV is less than $50,000/QALY and $100,000/QALY with a probability of 44.48% and 97.24%, respectively.

CONCLUSION

Although adjuvant chemotherapy with 5FU/LV is cost-effective at all ages for patients who have undergone an uncomplicated hemicolectomy for stage II colon cancer, FOLFOX is not likely to be cost-effective as compared with 5FU/LV.

Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-2

The appropriate development of a model begins with understanding the problem that is being represented. The aim of this article is to provide a series of consensus-based best practices regarding the process of model conceptualization. For the purpose of this series of papers, the authors consider the development of models whose purpose is to inform medical decisions and health-related resource allocation questions. They specifically divide the conceptualization process into two distinct components: the conceptualization of the problem, which converts knowledge of the health care process or decision into a representation of the problem, followed by the conceptualization of the model itself, which matches the attributes and characteristics of a particular modeling type to the needs of the problem being represented. Recommendations are made regarding the structure of the modeling team, agreement on the statement of the problem, the structure, perspective and target population of the model, and the interventions and outcomes represented. Best practices relating to the specific characteristics of model structure, and which characteristics of the problem might be most easily represented in a specific modeling method, are presented. Each section contains a number of recommendations that were iterated among the authors, as well as the wider modeling taskforce, jointly set up by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making.

Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--2

The appropriate development of a model begins with understanding the problem that is being represented. The aim of this article was to provide a series of consensus-based best practices regarding the process of model conceptualization. For the purpose of this series of articles, we consider the development of models whose purpose is to inform medical decisions and health-related resource allocation questions. We specifically divide the conceptualization process into two distinct components: the conceptualization of the problem, which converts knowledge of the health care process or decision into a representation of the problem, followed by the conceptualization of the model itself, which matches the attributes and characteristics of a particular modeling type with the needs of the problem being represented. Recommendations are made regarding the structure of the modeling team, agreement on the statement of the problem, the structure, perspective, and target population of the model, and the interventions and outcomes represented. Best practices relating to the specific characteristics of model structure and which characteristics of the problem might be most easily represented in a specific modeling method are presented. Each section contains a number of recommendations that were iterated among the authors, as well as among the wider modeling taskforce, jointly set up by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making.

Design and validation of a data simulation model for longitudinal healthcare data

Evaluating performance characteristics of analytic methods developed to identify treatment effects in longitudinal healthcare data has been hindered by lack of an objective benchmark to measure performance. Relationships between drugs and subsequent treatment effects are not precisely quantified in real-world data, and simulated data offer potential to augment method development by providing data with known, measurable characteristics. However, the use of simulated data has been limited due to its inability to adequately reflect the complexities inherent in real-world databases that are necessary for effective method development. The goal of this study was to develop and evaluate a model for simulating longitudinal healthcare data that adequately captures these complexities. An empiric design was chosen that utilizes the characteristics of a real healthcare database as simulation input. This model demonstrates the potential for simulated data with known characteristics to adequately reflect complex relationships among diseases and treatments as recorded in healthcare databases.

Accounting for Methodological, Structural, and Parameter Uncertainty in Decision-Analytic Models

Accounting for uncertainty is now a standard part of decision-analytic modeling and is recommended by many health technology agencies and published guidelines. However, the scope of such analyses is often limited, even though techniques have been developed for presenting the effects of methodological, structural, and parameter uncertainty on model results. To help bring these techniques into mainstream use, the authors present a step-by-step guide that offers an integrated approach to account for different kinds of uncertainty in the same model, along with a checklist for assessing the way in which uncertainty has been incorporated. The guide also addresses special situations such as when a source of uncertainty is difficult to parameterize, resources are limited for an ideal exploration of uncertainty, or evidence to inform the model is not available or not reliable. Methods for identifying the sources of uncertainty that influence results most are also described. Besides guiding analysts, the guide and checklist may be useful to decision makers who need to assess how well uncertainty has been accounted for in a decision-analytic model before using the results to make a decision.