On criteria for evaluating models of absolute risk

Evaluating the predictiveness of a continuous marker

Consider a continuous marker for predicting a binary outcome. For example, the serum concentration of prostate specific antigen may be used to calculate the risk of finding prostate cancer in a biopsy. In this article, we argue that the predictive capacity of a marker has to do with the population distribution of risk given the marker and suggest a graphical tool, the predictiveness curve, that displays this distribution. The display provides a common meaningful scale for comparing markers that may not be comparable on their original scales. Some existing measures of predictiveness are shown to be summary indices derived from the predictiveness curve. We develop methods for making inference about the predictiveness curve, for making pointwise comparisons between two curves, and for evaluating covariate effects. Applications to risk prediction markers in cancer and cystic fibrosis are discussed.

C-reactive protein and the prediction of cardiovascular events among those at intermediate risk: moving an inflammatory hypothesis toward consensus

Over 20 large-scale prospective studies show that the inflammatory biomarker high-sensitivity C-reactive protein (hsCRP) is an independent predictor of future cardiovascular events that additionally predicts risk of incident hypertension and diabetes. In many studies, the relative impact of hsCRP is at least as large as that individually of low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, blood pressure, or smoking, and knowledge of hsCRP correctly reclassifies a substantial proportion of "intermediate-risk" individuals into clinically relevant higher- or lower-risk categories. Other studies show the relative benefit of statins to be greater among those with increased hsCRP and that achieved hsCRP levels after statin therapy predict recurrent event rates as much as achieved levels of low-density lipoprotein cholesterol. Nonetheless, it remains controversial whether the time has come to modify traditional algorithms used for global risk detection. As described here, 6 areas of controversy regarding hsCRP are resolvable with a consensus position that focuses in primary prevention on selective use among individuals with 5% to 20% 10-year risk as estimated by Adult Treatment Panel III, and focuses in secondary prevention on high-risk patients being treated with statin therapy. Forthcoming trial data could expand or contract this "screen selectively" policy, and investigators should be open to the possibility that second-generation inflammatory biomarkers may be developed that supplant hsCRP altogether. In the meantime, however, this consensus position on hsCRP should be one to which both advocates and critics of the inflammatory hypothesis of atherosclerosis can adhere because it is one that can immediately improve patient care.

Use and misuse of the receiver operating characteristic curve in risk prediction

The c statistic, or area under the receiver operating characteristic (ROC) curve, achieved popularity in diagnostic testing, in which the test characteristics of sensitivity and specificity are relevant to discriminating diseased versus nondiseased patients. The c statistic, however, may not be optimal in assessing models that predict future risk or stratify individuals into risk categories. In this setting, calibration is as important to the accurate assessment of risk. For example, a biomarker with an odds ratio of 3 may have little effect on the c statistic, yet an increased level could shift estimated 10-year cardiovascular risk for an individual patient from 8% to 24%, which would lead to different treatment recommendations under current Adult Treatment Panel III guidelines. Accepted risk factors such as lipids, hypertension, and smoking have only marginal impact on the c statistic individually yet lead to more accurate reclassification of large proportions of patients into higher-risk or lower-risk categories. Perfectly calibrated models for complex disease can, in fact, only achieve values for the c statistic well below the theoretical maximum of 1. Use of the c statistic for model selection could thus naively eliminate established risk factors from cardiovascular risk prediction scores. As novel risk factors are discovered, sole reliance on the c statistic to evaluate their utility as risk predictors thus seems ill-advised.

Gail model for prediction of absolute risk of invasive breast cancer: independent evaluation in the Florence-European Prospective Investigation Into Cancer and Nutrition cohort

BACKGROUND

The Gail model 2 (GM) for predicting the absolute risk of invasive breast cancer has been used for counseling and to design intervention studies. Although the GM has been validated in US populations, its performance in other populations is unclear because of the wide variation in international breast cancer rates.

METHODS

We used data from a multicenter case-control study in Italy and from Italian cancer registries to develop a model (IT-GM) that uses the same risk factors as the GM. We evaluated the accuracy of the IT-GM and the GM using independent data from the Florence-European Prospective Investigation Into Cancer and Nutrition (EPIC) cohort. To assess model calibration (i.e., how well the model predicts the observed numbers of events in subsets of the population), we compared the number of expected incident breast cancers (E) predicted by these models with the number of observed incident breast cancers (O), and we computed the concordance statistic to measure discriminatory accuracy.

RESULTS

The overall E/O ratios were 0.96 (95% confidence interval [CI] = 0.84 to 1.11) and 0.93 (95% CI = 0.81 to 1.08) for the IT-GM and the GM, respectively. The IT-GM was somewhat better calibrated than GM in women younger than 50 years, but the GM was better calibrated when age at first live birth categories were considered (e.g., 20- to 24-year age-at-first-birth category E/O = 0.68, 95% CI = 0.53 to 0.94 for the IT-GM and E/O = 0.75, 95% CI = 0.58 to 1.03 for the GM). The concordance statistic was approximately 59% for both models, with 95% confidence intervals indicating that the models perform statistically significantly better than pure chance (concordance statistic of 50%).

CONCLUSIONS

There was no statistically significant evidence of miscalibration overall for either the IT-GM or the GM, and the models had equivalent discriminatory accuracy. The good performance of the IT-GM when applied on the independent data from the Florence-EPIC cohort indicates that GM can be improved for use in populations other than US populations. Our findings suggest that the Italian data may be useful for revising the GM to include additional risk factors for breast cancer.

Projecting absolute invasive breast cancer risk in white women with a model that includes mammographic density

BACKGROUND

To improve the discriminatory power of the Gail model for predicting absolute risk of invasive breast cancer, we previously developed a relative risk model that incorporated mammographic density (DENSITY) from data on white women in the Breast Cancer Detection Demonstration Project (BCDDP). That model also included the variables age at birth of first live child (AGEFLB), number of affected mother or sisters (NUMREL), number of previous benign breast biopsy examinations (NBIOPS), and weight (WEIGHT). In this study, we developed the corresponding model for absolute risk.

METHODS

We combined the relative risk model with data on the distribution of the variables AGEFLB, NUMREL, NBIOPS, and WEIGHT from the 2000 National Health Interview Survey, with data on the conditional distribution of DENSITY given other risk factors in BCDDP, with breast cancer incidence rates from the Surveillance, Epidemiology, and End Results program of the National Cancer Institute, and with national mortality rates. Confidence intervals (CIs) accounted for variability of estimates of relative risks and of risk factor distributions. We compared the absolute 5-year risk projections from the new model with those from the Gail model on 1744 white women.

RESULTS

Attributable risks of breast cancer associated with DENSITY, AGEFLB, NUMREL, NBIOPS, and WEIGHT were 0.779 (95% CI = 0.733 to 0.819) and 0.747 (95% CI = 0.702 to 0.788) for women younger than 50 years and 50 years or older, respectively. The model predicted higher risks than the Gail model for women with a high percentage of dense breast area. However, the average risk projections from the new model in various age groups were similar to those from the Gail model, suggesting that the new model is well calibrated.

CONCLUSIONS

This new model for absolute invasive breast cancer risk in white women promises modest improvements in discriminatory power compared with the Gail model but needs to be validated with independent data.

Validation of a model of lung cancer risk prediction among smokers

The Bach model was developed to predict the absolute 10-year risk of developing lung cancer among smokers by use of participants in the Carotene and Retinol Efficacy Trial of lung cancer prevention. We assessed the validity of the Bach model among 6239 smokers from the placebo arm of the Alpha-Tocopherol, Beta-Carotene Cancer Prevention (ATBC) Study. The expected numbers of lung cancer cases and deaths without lung cancer were calculated from the Bach model and compared with the observed numbers of corresponding events over 10 years. We found that the risk model slightly underestimated the observed lung cancer risk (number of lung cancers expected/number observed = 0.89, 95% confidence interval [CI] = 0.80 to 0.99) over 10 years. The competing risk portion of the model substantially underestimated risk of non-lung cancer mortality (number of non-lung cancer deaths expected/number observed = 0.61, 95% CI = 0.57 to 0.64) over 10 years. The age-specific concordance indices for 10-year predictions were 0.77 (95% CI = 0.70 to 0.84), 0.59 (95% CI = 0.53 to 0.65), 0.62 (95% CI = 0.57 to 0.67), and 0.57 (95% CI = 0.49 to 0.67) for the age groups 50-54, 55-59, 60-64, and 65-69 years, respectively. Periodic radiographic screening in the ATBC Study may explain why slightly more cancers were observed than expected from the Bach model.