TECH-VER: A Verification Checklist to Reduce Errors in Models and Improve Their Credibility

BACKGROUND

In health economic literature, checklists or best practice recommendations on model validation/credibility always declare verification of the programmed model as a fundamental step, such as 'is the model implemented correctly and does the implementation accurately represent the conceptual model?' However, to date, little operational guidance for the model verification process has been given. In this study, we aimed to create an operational checklist for model users or reviewers to verify the technical implementation of health economic decision analytical models and document their verification efforts.

METHODS

Literature on model validation, verification, programming errors and credibility was reviewed systematically from scientific databases. An initial beta version of the checklist was developed based on the checklists/tests identified from the literature and from authors' previous modeling/appraisal experience. Next, the first draft checklist was presented to a number of health economists on several occasions and was tested on different models (built in different software, developed by different stakeholders, including drug manufacturers, consultancies or academia), each time leading to an update of the checklist and culminating in the final version of the TECHnical VERification (TECH-VER) checklist, introduced in this paper.

RESULTS

The TECH-VER necessitates a model reviewer (preferably independent), an executable and transparent model, its input sources, and detailed documentation (e.g. technical report/scientific paper) in which the conceptual model, its implementation, programmed model inputs, and results are reported. The TECH-VER checklist consists of five domains: (1) input calculations; (2) event-state (patient flow) calculations; (3) result calculations; (4) uncertainty analysis calculations; and (5) other overall checks (e.g. validity or interface). The first four domains reflect the verification of the components of a typical health economic model. For these domains, as a prerequisite of verification tests, the reviewer should identify the relevant calculations in the electronic model and assess the provided justifications for the methods used in the identified calculations. For this purpose, we recommend completeness/consistency checks. Afterwards, the verification tests can be conducted for the calculations in each of these stages by checking the correctness of the implementation of these calculations. For this purpose, the following type of tests are recommended in consecutive order: (i) black-box tests, i.e. checking if model calculations are in line with a priori expectations; (ii) white-box testing, i.e. going through the program code details line by line, or cell by cell (recommended for some crucial calculations and if there are some unexpected results from the black-box tests); and (iii) model replication/parallel programming (recommended only in certain situations, and if the issues related to the identified unexpected results from black-box tests could not be resolved through white-box testing). To reduce the time burden of model verification, we suggest a hierarchical order in tests i-iii, where going to the next step is necessary when the previous step fails.

CONCLUSIONS

The TECH-VER checklist is a comprehensive checklist for the technical verification of decision analytical models, aiming to help identify model implementation errors and their root causes while improving the transparency and efficiency of the verification efforts. In addition to external reviews, we consider that the TECH-VER can be used as an internal training and quality control tool for new health economists, while developing their initial models. It is the authors' aim that the TECH-VER checklist transforms itself to an open-source living document, with possible future versions, or 'bolt-on' extensions for specific applications with additional 'fit-for-purpose' tests, as well as 'tips and tricks' and some demonstrative error examples. For this reason, the TECH-VER checklist and the list of black-box tests created in this paper and a few model verification examples is uploaded to an open access, online platform (github and the website of the institute), where other users will also be able to upload their original verification efforts and tests.

Obinutuzumab in Combination with Chemotherapy for the First-Line Treatment of Patients with Advanced Follicular Lymphoma : An Evidence Review Group Evaluation of the NICE Single Technology Appraisal

The National Institute for Health and Care Excellence (NICE), as part of the institute's single technology appraisal (STA) process, invited the company that makes obinutuzumab (Roche Products Limited) to submit evidence of the clinical and cost effectiveness of the drug in combination with chemotherapy, with or without obinutuzumab as maintenance therapy for adult patients with untreated, advanced follicular lymphoma (FL) in the UK. Kleijnen Systematic Reviews Ltd (KSR), in collaboration with Erasmus University Rotterdam, was commissioned to act as the Evidence Review Group (ERG). This paper describes the company's submission, the ERG review, and NICE's subsequent decisions. The clinical evidence was derived from two phase III, company-sponsored, randomised, open-label studies. Most evidence on obinutuzumab was based on the GALLIUM trial that compared obinutuzumab in combination with chemotherapy as induction followed by obinutuzumab maintenance monotherapy with rituximab in combination with chemotherapy as induction followed by rituximab maintenance monotherapy in previously untreated patients with FL (grades 1-3a). Long-term clinical evidence was based on the PRIMA trial, studying the benefit of two years of rituximab maintenance after first-line treatment in patients with FL. The cost-effectiveness evidence submitted by the company relied on a partitioned survival cost-utility model, implemented in Microsoft® Excel. The base-case incremental cost-effectiveness ratio (ICER) presented in the company submission was <£20,000 per quality-adjusted life-year (QALY) gained. Although the ERG concluded that the economic model met the NICE reference case to a reasonable extent, some errors were identified and several assumptions made by the company were challenged. A new base-case scenario produced by the ERG suggested an ICER that was higher than the company base case, but still below £30,000 per QALY gained. However, some ERG scenario analyses were close to or even above the threshold. This was the case in particular for assuming a treatment effect that did not extend beyond trial follow-up. These results led to an initial negative recommendation by the appraisal committee. Subsequently, the company submitted a revised base case focusing on patients at intermediate or high risk of premature mortality. Simultaneously, a further price discount for obinutuzumab was granted. In addition to the company's revised base case, the ERG suggested a restriction of the treatment effect to 5 years and implemented biosimilar uptake and cheaper prices for rituximab. All of these adjustments did not exceed £30,000 per QALY gained and therefore the use of obinutuzumab for patients with advanced FL and a Follicular Lymphoma International Predictive Index (FLIPI) score of two or more could be recommended.

Ribociclib with an Aromatase Inhibitor for Previously Untreated, HR-Positive, HER2-Negative, Locally Advanced or Metastatic Breast Cancer: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

The National Institute for Health and Care Excellence, as part of the institute's single technology appraisal process, invited the manufacturer of ribociclib (Kisqali^®, Novartis) to submit evidence regarding the clinical and cost effectiveness of the drug in combination with an aromatase inhibitor for the treatment of previously untreated, hormone receptor-positive, human epidermal growth factor receptor 2-negative, locally advanced or metastatic breast cancer. Kleijnen Systematic Reviews Ltd and Erasmus University Rotterdam were commissioned as the Evidence Review Group for this submission. The Evidence Review Group reviewed the evidence submitted by the manufacturer, corrected and validated the manufacturer's decision analytic model, and conducted exploratory analyses to assess the robustness and validity of the presented clinical and cost-effectiveness results. This article describes the company submission, the Evidence Review Group assessment and National Institute for Health and Care Excellence subsequent decisions. The main clinical effectiveness evidence was obtained from the MONALEESA-2 trial, a randomised controlled trial comparing ribociclib plus letrozole with placebo plus letrozole. Progression-free survival was significantly longer in the ribociclib group (95% confidence interval, 19.3-not reached) vs. 14.7 months (95% confidence interval 13.0-16.5) in the placebo group. To assess the cost effectiveness of ribociclib in combination with an aromatase inhibitor, the company developed an individual patient-level model using a discrete-event simulation approach in Microsoft^® Excel. In the model, simulated patients move through a series of three health states until death, i.e. first-line progression-free survival, second-line progression-free survival and progressive disease. The length of progression-free survival during the first line was informed by the MONALEESA-2 trial. The benefit in progression-free survival in the first line was transferred to a benefit in overall survival assuming full progression-free survival to overall survival surrogacy (because of the immaturity of overall survival data from the MONALEESA-2 trial). Patient-level data from the BOLERO-2 trial, evaluating the addition of everolimus to exemestane in the second-line treatment of postmenopausal HR-positive advanced breast cancer, were used to inform the length of progression-free survival during the second line. Costs included in the model were treatment costs (e.g. technology acquisition costs of first, second, third and/or later line treatments), drug administration costs, monitoring costs and health state costs (including terminal care). Additionally, the costs of adverse events associated with the first-line treatment were incorporated. The Evidence Review Group recalculated the incremental cost-effectiveness ratio using data from a different data cut-off date from the MONALEESA-2 trial and by changing some assumptions (e.g. progression-free survival to overall survival surrogacy approach and post-progression third and/or later line treatment-related costs). After two appraisal committee meetings and a revised base case submitted by the company (including a second enhanced patient access scheme discount), the committee concluded that taking into account the uncertainties in the calculation of the cost effectiveness, there were plausible cost-effectiveness estimates broadly in the range that could be considered as a cost-effective use of National Health Service resources. Therefore, ribociclib was recommended as a treatment option for the first-line treatment of hormone receptor-positive, human epidermal growth factor receptor 2-negative breast cancer, contingent on the company providing ribociclib with the discount agreed in the second enhanced patient access scheme.

Pomalidomide with Dexamethasone for Treating Relapsed and Refractory Multiple Myeloma Previously Treated with Lenalidomide and Bortezomib: An Evidence Review Group Perspective of an NICE Single Technology Appraisal

The National Institute for Health and Care Excellence (NICE), as part of the institute's single technology appraisal (STA) process, invited the manufacturer of pomalidomide (POM; Imnovid^®, Celgene) to submit evidence regarding the clinical and cost effectiveness of the drug in combination with dexamethasone (POM + LoDEX) for the treatment of relapsed and refractory multiple myeloma (RRMM) after at least two regimens including lenalidomide (LEN) and bortezomib (BOR). Kleijnen Systematic Reviews Ltd (KSR) and Erasmus University Rotterdam were commissioned as the Evidence Review Group (ERG) for this submission. The ERG reviewed the evidence submitted by the manufacturer, validated the manufacturer's decision analytic model, and conducted exploratory analyses in order to assess the robustness and validity of the presented clinical and cost-effectiveness results. This paper describes the company submission, the ERG assessment, and NICE's subsequent decisions. The company conducted a systematic review to identify studies comparing POM with comparators outlined in the NICE scope: panobinostat with bortezomib and dexamethasone (PANO + BOR + DEX), bendamustine with thalidomide and dexamethasone (BTD) and conventional chemotherapy (CC). The main clinical effectiveness evidence was obtained from MM-003, a randomized controlled trial (RCT) comparing POM + LoDEX with high-dose dexamethasone (HiDEX; used as a proxy for CC). Additional data from other studies were also used as nonrandomized observational data sources for the indirect treatment comparison of POM + LoDEX with BTD and PANO + BOR + DEX. Covariate or treatment switching adjustment methods were used for each comparison. The model developed in Microsoft^® Excel 2010 using a semi-Markov partitioned survival structure, submitted in the original submission to NICE for TA338, was adapted for the present assessment of the cost effectiveness of POM + LoDEX. Updated evidence from the clinical-effectiveness part was used for the survival modelling of progression-free survival and overall survival. For POM + LoDEX, the patient access scheme (PAS) discount was applied to the POM price. Three separate comparisons were conducted for each comparator, each comparison using a different dataset and adjustment methods. The ERG identified and corrected some errors, and the corrected incremental cost-effectiveness ratios (ICERs) for POM + LoDEX versus each comparator were presented: approximately £45,000 per quality-adjusted life-year (QALY) gained versus BTD, savings of approximately £143,000 per QALY lost versus PANO + BOR + DEX, and approximately £49,000 per QALY gained versus CC. The ERG also conducted full incremental analyses, which revealed that CC, POM + LoDEX and PANO + BOR + DEX were on the cost-effectiveness frontier. The committee's decision on the technology under analysis deemed that POM + LoDEX should be recommended as an option for treating multiple myeloma in adults at third or subsequent relapse of treatments including both LEN and BOR, contingent on the company providing POM with the discount agreed in the PAS.

Ramucirumab for Treating Advanced Gastric Cancer or Gastro-Oesophageal Junction Adenocarcinoma Previously Treated with Chemotherapy: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

The National Institute for Health and Care Excellence (NICE) invited the company that manufactures ramucirumab (Cyramza®, Eli Lilly and Company) to submit evidence of the clinical and cost effectiveness of the drug administered alone (monotherapy) or with paclitaxel (combination therapy) for treating adults with advanced gastric cancer or gastro-oesophageal junction (GC/GOJ) adenocarcinoma that were previously treated with chemotherapy, as part of the Institute's single technology appraisal (STA) process. Kleijnen Systematic Reviews Ltd (KSR), in collaboration with Erasmus University Rotterdam, was commissioned to act as the Evidence Review Group (ERG). This paper describes the company's submission, the ERG review, and NICE's subsequent decisions. Clinical effectiveness evidence for ramucirumab monotherapy (RAM), compared with best supportive care (BSC), was based on data from the REGARD trial. Clinical effectiveness evidence for ramucirumab combination therapy (RAM + PAC), compared with paclitaxel monotherapy (PAC), was based on data from the RAINBOW trial. In addition, the company undertook a network meta-analysis (NMA) to compare RAM + PAC with BSC and docetaxel. Cost-effectiveness evidence of monotherapy and combination therapy relied on partitioned survival, cost-utility models. The base-case incremental cost-effectiveness ratio (ICER) of the company was £188,640 (vs BSC) per quality-adjusted life-year (QALY) gained for monotherapy and £118,209 (vs BSC) per QALY gained for combination therapy. The ERG assessment indicated that the modelling structure represented the course of the disease; however, a few errors were identified and some of the input parameters were challenged. The ERG provided a new base case, with ICERs (vs BSC) of £188,100 (monotherapy) per QALY gained and £129,400 (combination therapy) per QALY gained and conducted additional exploratory analyses. The NICE Appraisal Committee (AC), considered the company's decision problem was in line with the NICE scope, with the exception of the choice of comparators for the combination therapy model. The most plausible ICER for ramucirumab monotherapy compared with BSC was £188,100 per QALY gained. The Committee considered that the ERG's exploratory analysis in which RAM + PAC was compared with PAC by using the direct head-to-head data (including utilities) from the RAINBOW trial, provided the most plausible ICER (i.e. £408,200 per QALY gained) for ramucirumab combination therapy. The Committee concluded that end-of-life considerations cannot be applied for either case, since neither failed to offer an extension to life of at least 3 months. The company did not submit a patient access scheme (PAS). After consideration of the evidence, the Committee concluded that ramucirumab alone or with paclitaxel could not be considered a cost-effective use of National Health Service resources for treating advanced GC/GOJ patients that were previously treated with chemotherapy, and therefore its use could not be recommended. We might wonder if a complete STA process is necessary for treatments without a PAS, which are, according to the company's submission, already associated with ICERs far above the currently accepted threshold in all (base-case, sensitivity and scenario) analyses.