Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer. In: Lyman GH, Burstein HJ, editors. Breast Cancer: Translational Therapeutic Strategies. Informa Healthcare; 2007. pp. 11-26. [DOI: 10.3109/9781420019940.002]

Prediction of risk of recurrence of venous thromboembolism following treatment for a first unprovoked venous thromboembolism: systematic review, prognostic model and clinical decision rule, and economic evaluation

BACKGROUND

Unprovoked first venous thromboembolism (VTE) is defined as VTE in the absence of a temporary provoking factor such as surgery, immobility and other temporary factors. Recurrent VTE in unprovoked patients is highly prevalent, but easily preventable with oral anticoagulant (OAC) therapy. The unprovoked population is highly heterogeneous in terms of risk of recurrent VTE.

OBJECTIVES

The first aim of the project is to review existing prognostic models which stratify individuals by their recurrence risk, therefore potentially allowing tailored treatment strategies. The second aim is to enhance the existing research in this field, by developing and externally validating a new prognostic model for individual risk prediction, using a pooled database containing individual patient data (IPD) from several studies. The final aim is to assess the economic cost-effectiveness of the proposed prognostic model if it is used as a decision rule for resuming OAC therapy, compared with current standard treatment strategies.

METHODS

Standard systematic review methodology was used to identify relevant prognostic model development, validation and cost-effectiveness studies. Bibliographic databases (including MEDLINE, EMBASE and The Cochrane Library) were searched using terms relating to the clinical area and prognosis. Reviewing was undertaken by two reviewers independently using pre-defined criteria. Included full-text articles were data extracted and quality assessed. Critical appraisal of included full texts was undertaken and comparisons made of model performance. A prognostic model was developed using IPD from the pooled database of seven trials. A novel internal-external cross-validation (IECV) approach was used to develop and validate a prognostic model, with external validation undertaken in each of the trials iteratively. Given good performance in the IECV approach, a final model was developed using all trials data. A Markov patient-level simulation was used to consider the economic cost-effectiveness of using a decision rule (based on the prognostic model) to decide on resumption of OAC therapy (or not).

RESULTS

Three full-text articles were identified by the systematic review. Critical appraisal identified methodological and applicability issues; in particular, all three existing models did not have external validation. To address this, new prognostic models were sought with external validation. Two potential models were considered: one for use at cessation of therapy (pre D-dimer), and one for use after cessation of therapy (post D-dimer). Model performance measured in the external validation trials showed strong calibration performance for both models. The post D-dimer model performed substantially better in terms of discrimination (c = 0.69), better separating high- and low-risk patients. The economic evaluation identified that a decision rule based on the final post D-dimer model may be cost-effective for patients with predicted risk of recurrence of over 8% annually; this suggests continued therapy for patients with predicted risks ≥ 8% and cessation of therapy otherwise.

CONCLUSIONS

The post D-dimer model performed strongly and could be useful to predict individuals' risk of recurrence at any time up to 2-3 years, thereby aiding patient counselling and treatment decisions. A decision rule using this model may be cost-effective for informing clinical judgement and patient opinion in treatment decisions. Further research may investigate new predictors to enhance model performance and aim to further externally validate to confirm performance in new, non-trial populations. Finally, it is essential that further research is conducted to develop a model predicting bleeding risk on therapy, to manage the balance between the risks of recurrence and bleeding.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42013003494.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist

Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.

Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines

Background:

Poor reporting compromises the reliability and clinical value of prognostic tumour marker studies. We review articles to assess the reporting of patients and events using REMARK guidelines, at the time of guideline publication.

Methods:

We sampled 50 prognostic tumour marker studies from higher impact cancer journals between 2006 and 2007. The inclusion criteria were cancer; focus on single biological tumour marker; survival analysis; multivariable analysis; and not gene array or proteomic data. Articles were assessed for the REMARK profile and other REMARK guideline items. We propose a reporting aid, the REMARK profile, motivated by the CONSORT flowchart.

Results:

In 50 studies assessed for the REMARK profile, the number of eligible patients (56% of articles), excluded patients (54%) and patients in analyses (98%) was reported. Only 50% of articles reported the number of outcome events. In multivariable analyses, 54% and 30% of articles reported patient and event numbers for all variables. Of the studies, 66% used archival samples, indicating a potentially biased patient selection. Only 36% of studies reported clearly defined outcomes.

Conclusions:

Good reporting is critical for the interpretability and clinical applicability of prognostic studies. Current reporting of key information, such as the number of outcome events in all patients and subgroups, is poor. Use of the REMARK profile would greatly improve reporting and enhance prognostic research.

Validation, updating and impact of clinical prediction rules: a review

OBJECTIVE

To provide an overview of the research steps that need to follow the development of diagnostic or prognostic prediction rules. These steps include validity assessment, updating (if necessary), and impact assessment of clinical prediction rules.

STUDY DESIGN AND SETTING

Narrative review covering methodological and empirical prediction studies from primary and secondary care.

RESULTS

In general, three types of validation of previously developed prediction rules can be distinguished: temporal, geographical, and domain validations. In case of poor validation, the validation data can be used to update or adjust the previously developed prediction rule to the new circumstances. These update methods differ in extensiveness, with the easiest method a change in model intercept to the outcome occurrence at hand. Prediction rules -- with or without updating -- showing good performance in (various) validation studies may subsequently be subjected to an impact study, to demonstrate whether they change physicians' decisions, improve clinically relevant process parameters, patient outcome, or reduce costs. Finally, whether a prediction rule is implemented successfully in clinical practice depends on several potential barriers to the use of the rule.

CONCLUSION

The development of a diagnostic or prognostic prediction rule is just a first step. We reviewed important aspects of the subsequent steps in prediction research.

A simple method to adjust clinical prediction models to local circumstances

Introduction

Clinical prediction models estimate the risk of having or developing a particular outcome or disease. Researchers often develop a new model when a previously developed model is validated and the performance is poor. However, the model can be adjusted (updated) using the new data. The updated model is then based on both the development and validation data. We show how a simple updating method may suffice to update a clinical prediction model.

Methods

A prediction model that preoperatively predicts the risk of severe postoperative pain was developed with multivariable logistic regression from the data of 1944 surgical patients in the Academic Medical Center Amsterdam, the Netherlands. We studied the predictive performance of the model in 1,035 new patients, scheduled for surgery at a later time in the University Medical Center Utrecht, the Netherlands. We assessed the calibration (agreement between predicted risks and the observed frequencies of an outcome) and discrimination (ability of the model to distinguish between patients with and without postoperative pain). When the incidence of the outcome is different, all predicted risks may be systematically over- or underestimated. Hence, the intercept of the model can be adjusted (updating).

Results

The predicted risks were systematically higher than the observed frequencies, corresponding to a difference in the incidence of postoperative pain between the development (62%) and validation set (36%). The updated model resulted in better calibration.

Discussion

When a clinical prediction model in new patients does not show adequate performance, an alternative to developing a new model is to update the prediction model with new data. The updated model will be based on more patient data, and may yield better risk estimates.